AWS Lake Formation Workshop

AWS Lake Formation Workshop > Handling Real-Time Data > Generate Streaming Data

Generate Streaming Data

In this section, you will generate real-time data using the Amazon Kinesis Data Generator tool configured previously.

Go Back to the console and use the username: kinesisgen and the password: kinesisgen1
Select the following options:

Go to the bottom part of the page and define the structure of our simulated data using the parametrized template. Copy and paste it to the Template1 tab. Template to use:

{
    "productName" : "{{commerce.productName}}",
    "color" : "{{commerce.color}}",
    "department" : "{{commerce.department}}",
    "product" : "{{commerce.product}}",
    "imageUrl": "{{image.imageUrl}}",
    "dateSoldSince": "{{date.past}}",
    "dateSoldUntil": "{{date.future}}",
    "price": {{random.number(
        {
            "min":10,
            "max":150
        }
    )}},
    "campaign": "{{random.arrayElement(
        ["BlackFriday","10Percent","NONE"]
    )}}"
}

KDG makes use of an open-source toolkit called Faker.js (available in GitHub: ). You can customize the template as you wish. Check Faker.js documentation for details.

Click on the Send Data button.
You will see a message similar to this:: Hint: Don’t stop sending events from KDG. Let it run while you are working on the other sections.
To verify that the Data Stream is working go back to the Kinesis Firehose console and select lf-kinesis-fh.
Select the Monitoring tab and wait 3 minutes and you will see how the metrics start to change, if you can see it, you have successfully created a Kinesis firehose data stream, and are ready to add this data to Lake Formation. Please proceed to the next lab.

In the next section, you are going to create a table for the real time data.