Nodejs Tutorial: Building a REST API With AWS SimpleDB and Node.js

Building the Server

To get started, let's install Express, a Node.js HTTP server framework:

npm install express —-save

Express manages most of the minutiae in setting up a server, but it doesn't include any facility for handling the HTTP request body, so we'll need to install another module, body-parser, to enable us to read the request body.

npm install body-parser --save

Body-parser has a few different options for parsing the body of the HTTP request. We’ll use the json() method for readability, but switching to another method is just swapping out the method on the bodyParser object. We only need the bodyParser method on the create and update methods, so we can just include it in those particular routes.

Create
Since each SimpleDB itemName needs to be unique, we can auto-generate a new itemName for each newly created item. We’re going to use the cuid module, which is a lightweight way to generate unique identifiers.

npm install cuid --save

SimpleDB expects attributes to be in the attribute name/value pair format:

[
{ "Name" : "attribute1", "Value" : "value1" },
{ "Name" : "attribute1", "Value" : "value2" },
{ "Name" : "attribute2", "Value" : "value3" },
{ "Name" : "attribute3", "Value" : "value4" }
]

Your server could certainly just accept and pass the values in this format directly to SimpleDB, but it is counter-intuitive to how data is often structured, and it's a difficult concept with which to work. We'll use a more intuitive data structure, an object/value array:

{
"attribute1" : ["value1","value2"],
"attribute2" : ["value3","value4"]
}

Here is a basic Express-based server with the create operation:

var
aws = require('aws-sdk'),
bodyParser = require('body-parser'),
cuid = require('cuid'),
express = require('express'),
sdbDomain = 'sdb-rest-tut',
app = express(),
simpledb;
aws.config.loadFromPath(process.env['HOME'] + '/aws.credentials.json');
simpledb = new aws.SimpleDB({
region : 'US-East',
endpoint : 'https://sdb.amazonaws.com'
});
//create
app.post(
'/inventory',
bodyParser.json(),
function(req,res,next) {
var
sdbAttributes = [],
newItemName = cuid();
//start with:
/*
{ attributeN : ['value1','value2',..'valueN'] }
*/
Object.keys(req.body).forEach(function(anAttributeName) {
req.body[anAttributeName].forEach(function(aValue) {
sdbAttributes.push({
Name : anAttributeName,
Value : aValue
});
});
});
//end up with:
/*
[
{ Name : 'attributeN', Value : 'value1' },
{ Name : 'attributeN', Value : 'value2' },
...
{ Name : 'attributeN', Value : 'valueN' },
]
*/
simpledb.putAttributes({
DomainName : sdbDomain,
ItemName : newItemName,
Attributes : sdbAttributes
}, function(err,awsResp) {
if (err) {
next(err); //server error to user
} else {
res.send({
itemName : newItemName
});
}
});
}
);
app.listen(3000, function () {
console.log('SimpleDB-powered REST server started.');
});

Let's start up your server and try it out. A great way to interact with a REST server is to use the cURL tool. This tool allows you to make an HTTP request with any verb right from the command line. To try out creating an item with our REST server, we'll need to activate a few extra options:

curl -H "Content-Type: application/json" -X POST -d '{"pets" : ["dog","cat"], "cars" : ["saab"]}' http://localhost:3000/inventory

After running the command, you'll see a JSON response with your newly created itemName or ID. If you switch over to SdbNavigator, you should see the new data when you query all the items.

Read

Now let’s build a basic function to read an item from SimpleDB. For this, we don’t need to perform a query since we’ll be getting the itemName or ID from the path of the request. We can perform a getAttributes request with that itemName or ID.

If we stopped here, we would have a functional but not very friendly form of our data. Let’s transform the Name/Value array into the same form we’re using to accept data (attribute : array of values). To accomplish this, we will need to go through each name/value pair and add it to a new array for each unique name.

Finally, let’s add the itemName and return the results.

//Read
app.get('/inventory/:itemID', function(req,res,next) {
simpledb.getAttributes({
DomainName : sdbDomain,
ItemName : req.params.itemID //this gets the value from :itemID in the path
}, function(err,awsResp) {
var
attributes = {};
if (err) {
next(err); //server error to users
} else {
awsResp.Attributes.forEach(function(aPair) {
// if this is the first time we are seeing the aPair.Name, let's add it to the response object, attributes as an array
if (!attributes[aPair.Name]) {
attributes[aPair.Name] = [];
}
//push the value into the correct array
attributes[aPair.Name].push(aPair.Value);
});
res.send({
itemName : req.params.itemID,
inventory : attributes
});
}
});
});

To test this, we need to use curl again. Try replacing [cuid] with the itemName or ID returned from our example of creating an item earlier in this tutorial.

curl -D- http://localhost:3000/inventory/[cuid]

Notice that we're using the -D- option. This will dump the HTTP head so we can see the response code.

Another aspect of REST is to use your response codes meaningfully. In the current example, if you supply a non-existent ID to curl, the above server will crash because you’re trying to forEach a non-existent array. We need to account for this and return a meaningful HTTP response code indicating that the item was not found.

To prevent the error, we should test for the existence of the variable awsResp.Attributes. If it doesn’t exist, let’s set the status code to 404 and end the http request. If it exists, then we can serve the response with attributes.

app.get('/inventory/:itemID', function(req,res,next) {
simpledb.getAttributes({
DomainName : sdbDomain,
ItemName : req.params.itemID
}, function(err,awsResp) {
var
attributes = {};
if (err) {
next(err);
} else {
if (!awsResp.Attributes) {
//set the status response to 404 because we didn't find any attributes then end it
res.status(404).end();
} else {
awsResp.Attributes.forEach(function(aPair) {
if (!attributes[aPair.Name]) {
attributes[aPair.Name] = [];
}
attributes[aPair.Name].push(aPair.Value);
});
res.send({
itemName : req.params.itemID,
inventory : attributes
});
}
}
});
});

Try it out with the new code and a non-existent ID and you'll see that the server returns a 404.

Now that we know how to use status to change the value, we should also update how we are responding to a POST/create. While the 200 response is technically correct as it means ‘OK’, a more insightful response code would be 201, which indicates ‘created’. To make this change, we’ll add it in the status method before sending.

res
.status(201)
.send({
itemName : newItemName
});

Update

Update is usually the most difficult operation for any system, and this REST server is no exception.

The nature of SimpleDB makes this operation a little more challenging as well. In the case of a REST server, an update is where you are replacing the entire piece of stored data; SimpleDB on the other hand, represents individual attribute/value pairs under an itemName.

To allow for an update to represent a single piece of data rather than a collection of name/value pairs, we need to define a schema for the purposes of our code (even though SimpleDB doesn’t need one). Don’t worry if this is unclear right now—keep reading and I’ll illustrate the requirement.

Compared to many other database systems, our schema will be very simple: just a defined array of attributes. For our example, we have four fields we are concerned with: pets, cars, furniture, and phones:

schema = ['pets','cars','furniture','phones'],

With SimpleDB you can’t store an empty attribute/value pair, nor does SimpleDB have any concept of individual items, so we’ll assume that if SimpleDB doesn’t return a value, it doesn’t exist. Similarly, if we try to update a SimpleDB item with an empty attribute/value pair, it will ignore that data. Take, for example, this data:

{
"itemName": "cil89uvnm00011ma2fykmy79c",
"inventory": {
"cars": [],
"pets": [
"cat",
"dog"
]
}
}

Logically, we know that cars, being an empty array, should have no values, and pets should have two values, but what about phones and furniture? What do you do to those? Here is how we translate this update request to work with SimpleDB:

Put an attribute pet with a value to cat.
Put an attribute pet with a value to dog.
Delete attributes for cars.
Delete attributes for phones.
Delete attributes for furniture.

Without some form of schema that at least defines the attributes, we wouldn’t know that phones and furniture needed to be deleted. Luckily, we can consolidate this update operation into two SimpleDB requests instead of five: one to put the attributes, and one to delete the attributes. This is a good time to pull out the code from the post/create function that transforms the attribute/array of values object into the attribute/value pair array.

function attributeObjectToAttributeValuePairs(attrObj, replace) {
var
sdbAttributes = [];
Object.keys(attrObj).forEach(function(anAttributeName) {
attrObj[anAttributeName].forEach(function(aValue) {
sdbAttributes.push({
Name : anAttributeName,
Value : aValue,
Replace : replace //if true, then SimpleDB will overwrite rather than append more values to an attribute
});
});
});
return sdbAttributes;
}

We’re going to make an important alteration to the create function as well. We’ll be adding a new attribute/value to all items. This attribute will not be added to the schema and is effectively read-only.

We will add an attribute called created and set the value to 1. With SimpleDB, there is limited ability to check if an item exists prior to adding attributes and values. On every putAttributes request you can check for the value and existence of a single attribute—in our case, we’ll use created and check for a value of 1. While this may seem like a strange workaround, it provides a very important safety to prevent the update operation from being able to create new items with an arbitrary ID.

newAttributes.push({
Name : 'created',
Value : '1'
});

Since we’ll be doing a couple of asynchronous HTTP requests, let’s install the async module to ease the handling of those callbacks.

npm install async —-save

Remember, since SimpleDB is distributed, there is no reason to sequentially put our attributes and then delete. We’ll use the function async.parallel to run these two operations and get a callback when both have completed. The responses from AWS form putAttributes and deleteAttributes do not provide important information, so we will just send an empty response with a status code 200 if there are no errors.

app.put(
'/inventory/:itemID',
bodyParser.json(),
function(req,res,next) {
var
updateValues = {},
deleteValues = [];
schema.forEach(function(anAttribute) {
if ((!req.body[anAttribute]) || (req.body[anAttribute].length === 0)) {
deleteValues.push({ Name : anAttribute});
} else {
updateValues[anAttribute] = req.body[anAttribute];
}
});
async.parallel([
function(cb) {
//update anything that is present
simpledb.putAttributes({
DomainName : sdbDomain,
ItemName : req.params.itemID,
Attributes : attributeObjectToAttributeValuePairs(updateValues,true),
Expected : {
Name : 'created',
Value : '1',
Exists : true
}
},
cb
);
},
function(cb) {
//delete any attributes that not present
simpledb.deleteAttributes({
DomainName : sdbDomain,
ItemName : req.params.itemID,
Attributes : deleteValues
},
cb
);
}
],
function(err) {
if (err) {
next(err);
} else {
res.status(200).end();
}
}
);
}
);

To take this for a spin, let's update a previously created entry. This time, we will make the inventory only include a "dog", removing all other items. Again, with cURL, run the command, substituting [cuid] with one of your item IDs.

curl -H "Content-Type: application/json" -X PUT -d '{"pets" : ["dog"] }' http://localhost:3000/inventory/[cuid]

Delete

SimpleDB has no concept of an item deletion, but it can delete attributes, as mentioned above. To delete an item, we’ll need to delete all the attributes and the ‘item' will cease to be.

Since we’ve defined a list of attributes in our schema, we’ll use the deleteAttributes call to remove all of those attributes as well as the created attribute. As per our plan, this operation will be at the same path as Update, but using the verb delete.

app.delete(
'/inventory/:itemID',
function(req,res,next) {
var
attributesToDelete;
attributesToDelete = schema.map(function(anAttribute){
return { Name : anAttribute };
});
attributesToDelete.push({ Name : 'created' });
simpledb.deleteAttributes({
DomainName : sdbDomain,
ItemName : req.params.itemID,
Attributes : attributesToDelete
},
function(err) {
if (err) {
next(err);
} else {
res.status(200).end();
}
}
);
}
);

List

Rounding out our REST verbs is list. To achieve the list operation, we’re going to use the select command and the SQL-like query language. Our list function will be barebones, but will serve as a good basis for more complex retrieval later on. We’re going to make a very simple query:

select * from `sdb-rest-tut` limit 100

As we ran into with the get/read operation, the response from SimpleDB isn’t very useful as it is focused on the attribute/value pairs. To avoid repeating ourselves, we’ll refactor the part of the get/read operation into a separate function and use it here. While we are at it, we’ll also filter out the created attribute (as it will show up in the get operation).

function attributeValuePairsToAttributeObject(pairs) {
var
attributes = {};
pairs
.filter(function(aPair) {
return aPair.Name !== 'created';
})
.forEach(function(aPair) {
if (!attributes[aPair.Name]) {
attributes[aPair.Name] = [];
}
attributes[aPair.Name].push(aPair.Value);
});
return attributes;
}

With a select operation, SimpleDB returns the values in the Items array. Each item is represented by an object that contains the itemName (as simply Name) and the attribute/value pairs.

To simplify this response, let’s return everything in a single object. First, we’ll convert the attribute/value pairs into an attribute/value array as we did in the read/get operation, and then we can add the itemName as the property ID.

app.get(
'/inventory',
function(req,res,next) {
simpledb.select({
SelectExpression : 'select * from `sdb-rest-tut` limit 100'
},
function(err,awsResp) {
var
items = [];
if (err) {
next(err);
} else {
items = awsResp.Items.map(function(anAwsItem) {
var
anItem;
anItem = attributeValuePairsToAttributeObject(anAwsItem.Attributes);
anItem.id = anAwsItem.Name;
return anItem;
});
res.send(items);
}
});
}
);

To see our results, we can use curl:

curl -D- -X GET http://localhost:3000/inventory

Validation

Validation is whole a subject of its own, but with the code we’ve already written, we have a start for a simple validation system.

For now, all we want to make sure is that a user can’t submit anything but what is in the schema. Looking back at the code that was written for update/put, forEaching over the schema will prevent any unauthorized attributes from being added, so we really just need to apply something similar to our create/post operation. In this case, we will filter the attribute/value pairs, eliminating any non-schema attributes.

newAttributes = newAttributes.filter(function(anAttribute) {
return schema.indexOf(anAttribute.Name) !== -1;
});

In your production code, you will likely want a more robust validation system. I would suggest integrating a JSON schema validator like ajv and building a middleware that sits between bodyParser and your route function on create and update operations.

Next Steps

With the code outlined in this article, you have all the operations needed to store, read and modify data, but this is only the start of your journey. In most cases, you’ll need to start thinking about the following topics: