Building a REST API With AWS SimpleDB and Node.js_part1

SimpleDB is a remote database offered by Amazon Web Services (AWS). The world of data stores is usually divided into SQL and NoSQL, based on the use (or non-use) of the SQL language. NoSQL data stores are usually based on a simpler key/value setup. SimpleDB straddles this line—it is a key/value store and it can also use a variant of SQL for retrieval. Most SQL languages are based on a schema that lays out the rows and columns of the data, but SimpleDB is a schema-less database, making for a very flexible data store.

In the SimpleDB database model, you have items, attributes and values.  Each row in the database is an item and can be identified by a unique and assignable item name. Each item can have up to 256 pairs of attributes and values. An unexpected aspect of SimpleDB is that an attribute can have more than one pair per item. I think the best way to think about SimpleDB is to think of a spreadsheet, but instead of each column/row intersection representing a single value, it represents an array of values.

This chart represents two items stored in a SimpleDB domain. The term domain is analogous to a “table” in other databases.

The first column is the item name—this is the only column where you can have only a single value, and you can think of it as a unique index column.

The other four columns (pets, cars, furniture, and phones) represent attributes that are currently in this domain—you aren’t limited to this, so every item can have an entirely unique set of attributes. In this data, the attribute pets on the item personInventory1 has three pairs; expressed in JSON, it’ll look something like this:

1. { "Name" : "pets", "Value" : "dog" }, 
2. { "Name" : "pets", "Value" : "cat" }, 
3. { "Name" : "pets", "Value" : "fish" }

On the other hand, the item personInventory2 has only one pair:

1. { "Name" : "pets", "Value" : "cat" }

While you don’t have to supply the same attributes for each item, you do need to supply at least one pair. This means that you cannot have an ‘empty’ item. Each attribute can have a value up to 1kb in size, so this means that each item is functionally limited to 256kb, due to the 1kb value limit and the 256 pair limit.

SimpleDB is distributed, which has some distinct traits that you need to understand and keep in mind as you design your app. Being a distributed database means a whole group of machines will respond to your requests and your data will be replicated throughout these servers. This distribution will be completely transparent to your program, but it does introduce the possibility of consistency issues—your data cannot be guaranteed to be present on all servers initially.

Don’t panic: it’s not as bad as it sounds for a few reasons. With SimpleDB, the consistency isn’t promised, but it is usually pretty good and quickly reaches all nodes from my experience. Designing around this also isn’t so hard—normally you try to avoid immediately reading a record you just wrote. Finally, SimpleDB has the option to perform consistent reads, but they are slower and may consume more resources. If your app requires consistent reading every time, you might want to reconsider using SimpleDB as your data store, but for many applications, this can be designed around or not even worried about.

On the upside, the distributed nature also affords SimpleDB a few advantages that mesh nicely with the Node.js environment. Since you don’t have a single server responding to your requests, you don’t need to worry about saturating the service, and you can achieve good performance by making many parallel requests to SimpleDB. Parallel and asynchronous requests are something that Node.js can handle easily.

Unlike many AWS services, there isn’t an Amazon-delivered console for management of SimpleDB. Luckily, there is a nice in-browser management console in the form of a Google Chrome plugin, SdbNavigator. In the SdbNavigator you can add or delete domains, insert, update and delete items, modify attributes, and perform queries.

AWS SDK

Now that we’ve gotten to know the SimpleDB service, let’s start writing our REST server. First, we’ll need to install the AWS SDK. This SDK handles not just SimpleDB but all the AWS services, so you may already be including it in your package.json file. To install the SDK, run the following from the command line:

1. npm install aws-sdk —-save

To use SimpleDB, you’ll also need to get your AWS credentials, which include an Access Key and a Secret Key. SimpleDB is a pay-as-you-go service, but AWS currently includes a generous free allowance for SimpleDB.

Word of warning: As with any pay-as-you-go service, be aware that it’s possible to write code that can rack up big bills, so you’re going to want to keep an eye on your usage and keep your credentials private and safe. 

Once you get the AWS SDK installed and have acquired your credentials, you’ll need to set up SimpleDB in your code. In this example, we'll use AWS credentials stored in a JSON file in your home directory. First, you’ll need to include the SDK module, create an AWS object, and finally set up your SimpleDB interface.

1. var
2.   aws         = require('aws-sdk'),
3.   simpledb;
4. aws.config.loadFromPath(process.env['HOME'] + '/aws.credentials.json');
5. //We'll use the Northern Virginia datacenter, change the region / endpoint for other datacenters http://docs.aws.amazon.com/general/latest/gr/rande.html#sdb_region
6. simpledb = new aws.SimpleDB({
7.   region    : 'US-East',
8.   endpoint  : 'https://sdb.amazonaws.com'
9. });

Notice that we are using a specific endpoint and region. Each datacenter is entirely independent, so if you create a Domain named “mysuperawesomedata” in Northern Virginia, it will not be replicated to nor present in the Sao Paulo datacenter, for example.

The SimpleDB object that you’ve created with new aws.SimpleDB is where all your methods for interacting with SimpleDB will be based. The AWS SDK for SimpleDB has only a few methods:

Batch Operations

• batchDeleteAttributes
• batchPutAttributes

Domain Management & Information

• createDomain
• deleteDomain
• domainMetadata
• listDomains

Item/Attribute Manipulation

• deleteAttributes
• getAttributes
• putAttributes

Querying

• select

In this tutorial, we will only be dealing with Item/Attribute Manipulation and Querying; while the other categories are useful, many applications will not have any use for them.

Test Data

Using SdbNavigator, enter your access and security keys into the tool, select ‘US-East’, and click connect.

Once you’ve successfully connected, let’s create a domain for testing. Click Add domain.

Then enter the domain name ‘sdb-rest-tut’ and click OK.

Now that you’ve created a domain, let’s enter some test data. Click Add property and add a property named “colors”. As a convention, I usually name properties in plural form to reflect the multi-value nature of SimpleDB.

Finally, we’ll click Add record to create our first SimpleDB item. In the ItemName() column, enter your unique item name. A quirk of SdbNavigator is that, by default, it will only accept a single value to each item, but this obscures the fact that a property can contain multiple values. To enter multiple values, click the S along the right edge of the property column.

In the new box, select Array to enter multiple values. In the Value column, enter “red”, and then click Add value and enter “blue”.

Finally, click Update to save the changes to this row.

Now that we’ve entered some test data, let’s make our first SimpleDB request from Node. We’ll just get everything in the Domain, which, at this point, will be just a single row.

1. var
2.   aws         = require('aws-sdk'),
3.   simpledb;
4.  
5. aws.config.loadFromPath(process.env['HOME'] + '/aws.credentials.json');
6.  
7. simpledb = new aws.SimpleDB({
8.   region        : 'US-East',
9.   endpoint  : 'https://sdb.amazonaws.com'
10. });
11.  
12. simpledb.select({
13.   SelectExpression  : 'select * from `sdb-rest-tut` limit 100'
14. }, function(err,resp) {
15.   if (err) {
16.     console.error(err);
17.   } else {
18.     console.log(JSON.stringify(resp,null,' '));
19.   }
20. });

The response will be logged to the console. Here is the response, annotated for explanation:

1. {
2.  "ResponseMetadata": {
3.   "RequestId": "...",             //Every request made to SimpleDB has a request ID
4.   "BoxUsage": "0.0000228616"      //This is how your account is charged, as of time of writing US-East region is 14 US cents per hour, so this request costs 0.00032 cents + the transfer cost (if you are currently outside of your free tier)
5.  },
6.  "Items": [                       //For a Select, your response will be in the "Items" object property
7.   {
8.    "Name": "myfirstitem",         //this is the itemName()
9.    "Attributes": [                //these are the attribute pairs
10.     {
11.      "Name": "colors",            //attribute name
12.      "Value": "red"               //value - note that every Value is a string, regardless of the contents
13.     },
14.     {
15.      "Name": "colors",            //Since the attribute name is repeated, we can see that `colors` has more than one value
16.      "Value": "blue"
17.     }
18.    ]
19.   }
20.  ]
21. }

A REST Server

Since we’ll be building a REST Server that stores data in SimpleDB, it’s important to understand what a REST server does. REST stands for REpresentational State Transfer. A REST server is really just a server that uses HTTP standard mechanisms as an interface for your data. Often, REST is used for server-to-server communications, but you can use REST servers with the client through JavaScript libraries such as jQuery or Angular. Generally, however, an end-user won’t interact directly with a REST server.

Interestingly, the AWS SDK actually uses the REST protocol to interact with SimpleDB, so it may seem odd to create a REST server to another REST server. You wouldn’t want to use the SimpleDB REST API directly because you need to authenticate your requests, which would risk the security of your AWS account. Also, by writing a server, you’ll be able to add a layer of both abstraction and validation to your data storage that will make the rest of your whole application much easier to deal with.

In this tutorial we will be building the basic CRUD+L functions, that is Create, Read, Update, Delete and List. If you think about it, you can break down most applications into CRUD+L. With REST, you will use a limited number of paths and several HTTP methods or verbs to create an intuitive API. Most developers are familiar with a few of the HTTP verbs, namely GET and POST, as they are used most often in web applications, but there are several others.

Notice that Read and List both use the same verb; we will be using slightly different paths to differentiate between the two. We’re using POST to represent Create as creating is not considered idempotent. Idempotent means that multiple identical calls will have the same result to the user and in your data, so an update (aka PUT) would be considered idempotent.

As our example, we’ll build a personal inventory server—a database to save whatever you own. Here is how the paths will look:

1234 is a placeholder for the person identifier (ID)—note that ‘create' and ‘list' do not have an ID. In the case of create, the ID will be generated, and with list, we’ll be getting all the names, so we don’t need a specific ID.
Written by Kyle Davis(continue)

If you found this post interesting, follow and support us.
Suggest for you:

How to Build a Slack Chatbot In Node.js using Botkit

Make A Real-Time Chat Room using Node Webkit, Socket.io, MEAN

Learn How To Deploy Node.Js App on Google Compute Engine

Learn and Understand NodeJS

Complete Node JS Developer Course Building 5 Real World Apps