{"componentChunkName":"component---src-templates-blog-post-js","path":"/get-youtube-data-aws","result":{"data":{"markdownRemark":{"html":"

\n

In this post, let's build a project to use AWS Lambda functions to load data from YouTube, and store it in an S3 bucket.

\n

This project can be found on GitHub as well.

\n

To make the project more interesting, let's define the problem we're trying to solve using YouTube data.

\n

What are the key factors that influence video engagement (likes, comments, and shares) and audience retention (watch time and drop-off rates) on YouTube, and how can creators optimize their content to maximize these metrics?

\n

The overall architecture of the solution we'll implement is outlined below.\nIn this post, we'll work on the first part of extracting raw data in to the S3 bucket.

\n

\"1-1\"

\n

It's worthwhile to have some idea about why we've chosen AWS Lambda and S3 among the many services available on the AWS platform.\nBeing serverless, Lambda functions are easy to build and deploy. Therefore, it's more suitable for a relatively shorter script to run as the one for getting and storing data. It also falls within the 15 minute maximum timeout value for Lambda functions.

\n

We're using an S3 bucket to store the raw data as it supports open data formats, and is scalable, durable, and cost-effective.

\n

1. Getting started with YouTube API

\n

The first step is to start using the YouTube API to get the data we are looking for using Python.

\n

YouTube has a helpful guide on getting started with the API.

\n

Create a new project in Google Developers Console. Name the project, I've called it as youtube-etl.

\n

\"aws-yt-2\"

\n

Create credentials for the project. Because we're not using private data, we're choosing to create an API key.

\n

\"aws-yt-3\"

\n

Head to API Console to enable the YouTube API.

\n

\"aws-yt-4\"

\n

Enable the YouTube Data API v3 for the project.

\n

\"aws-yt-5\"

\n

It should now appear in the list of enabled APIs.

\n

\n

Back in the Credentials page, restrict the API key we created to only use the YouTube API.

\n

\n

YouTube API has a daily limit of 10000 quota units. You can see this limit on Enabled APIs and Services > Youtube Data API -> Quotas and system limits.

\n

\"aws-yt-8\"

\n
Data Model for the dashboard
\n

The API Reference has the quota used for each request.

\n

\"aws-yt-9\"

\n

2. Using the YouTube API

\n

Before using the API, there're a few terms to get to know in order to use the API effectively.

\n

When an API request is sent, a list of items are returned. These items are referred to as resources.\nEach resource has groups of properties which are known as parts. See the list of parts available for video resource here.\nEach part contains a set of properties known as fields.

\n

When building your API request, you'll need to specify the part and field parameters, so that the response can be filtered. This helps to reduce latency with the request, and saves you from processing many properties which may not be used in your project.

\n

Let's try using curl to send a simple request to get the video categories.\nThe YouTube API has a videoCategories:List endpoint.

\n

You can use the API Explorer in the same page to help build your curl request. Click to view it in full screen mode.

\n

\"aws-yt-10\"

\n

Use snippet for the part parameter and US as the region code. The API has different region codes for different countries, and we'll have a look based on US in this example.

\n

\"aws-yt-11\"

\n

Run the curl command with your API Key in a terminal. You can remove the line for the access token as we aren't using any.

\n
curl -X GET 'https://youtube.googleapis.com/youtube/v3/videoCategories?part=snippet&regionCode=US&key=YOUR_API_KEY' --header 'Accept: application/json'
\n

The response will contain a list of video categories in list format.

\n

\n

3. Using the YouTube API to get data for the project

\n

As we now have some idea about using the API, let's try to build the request for the data we want for this project.

\n

The project aims to track the daily growth rate and audience engagement of popular travel videos, while identifying trending content. To achieve this, we'll focus on gathering popular travel videos published within the last 24 hours. To limit the scope, we'll focus on videos based on traveling in Japan.

\n

The YouTube API has a search endpoint which allows us to get a list of videos. It's worth noting that one search request will cost 100 quota units.

\n
curl -X GET \\\n'https://www.googleapis.com/youtube/v3/search?part=snippet&q=japan+travel&type=video&maxResults=50&publishedAfter=2024-12-11T00:00:00Z&publishedBefore=2024-12-12T00:00:00Z&order=viewCount&topic_id=/m/07bxq&videoDuration=medium&key=YOUR_API_KEY'
\n

Let's have a look at the parameters in the request.\nThe part parameter is set to return the snippet which will contain the video id.\nThe q parameter allows querying for travel videos.\nAs we want the videos published in the last 24 hours, the publishedAfter is set to the day before, and the publishedBefore to the next day, to limit the response to 24 hours.\nAs we're looking for the most popular videos, we're ordering the result by viewCount.\nThe topic Id helps to filter results to travel topic. Topic Ids are defined here.\nThe videoDuration parameter has 4 possible values - short (<4 minutes), medium (between 4 and 20 minutes), long (>20 minutes) and any (default value). We'll execute two requests with medium and long values to exclude shorts content on YouTube.

\n

The response contains a list of videos with their ids and other metadata.

\n

\n

4. Use Python to build the API request

\n

Now, let's use Python to build the above request.

\n

I've created a simple Python project in PyCharm, but you can use Jupyter Notebook as well.

\n

\n

Log in / Sign up to AWS Cloud Console. If you're creating a new account, you'll be able to use the free tier services.

\n

\n

In the AWS Console, search of Lambda service and click on Create a Function.\nAdd a function name and set the Runtime option to use Python. In addition to the function, an IAM role will be created with permission to write logs to the Amazon CloudWatch.

\n

\n

When you create the function, you'll see the code editor window where you can put in your code for the Lambda function. In the Runtime Settings section has the handler function defined as lambda_function.lambda_handler, which will be executed when the function is run.

\n

\"aws-yt-17\"

\n

Let's try running the sample function defined. Head to the Test tab and click on Test button to execute the function.\nThe function will execute and create CloudWatch logs. Expand to view the output and logs.

\n

\"aws-yt-18\"

\n

Let's head back to the IDE/Jupyter notebook and write the Python code to call the API.

\n
import os\nimport json\nfrom googleapiclient.discovery import build\n\n\ndef lambda_handler(event, context):\n    # get the API key from Lambda environment variables\n    youtube_api_key = os.environ['YOUTUBE_API_KEY']\n\n    # define api variables\n    api_name = 'youtube'\n    api_version = 'v3'\n\n    # initialise the API client\n    youtube = build(api_name, api_version, developerKey=youtube_api_key)\n\n    # execute search request for 50 videos with the highest view count\n    search_response = youtube.search().list(\n        part='id',\n        q='japan + travel',\n        type='video',\n        maxResults=50,\n        publishedAfter='2024-12-11T00:00:00Z',\n        publishedBefore='2024-12-12T00:00:00Z',\n        order='viewCount',\n        topicId='/m/07bxq',\n        videoDuration='medium'\n    ).execute()\n\n    # get video ids\n    video_ids = [item['id']['videoId'] for item in search_response.get('items', [])]\n\n    # get data for each video\n    video_data_response = youtube.videos().list(\n        part='snippet,content_details,statistics',\n        id=','.join(video_ids)\n    ).execute()\n\n    # extract video data\n    videos = []\n    for item in video_data_response.get('items', []):\n        # get() function allows a fallback value in case the element is not found in the video_data_response\n        snippet = item.get('snippet', {})\n        content_details = item.get('contentDetails', {})\n        statistics = item.get('statistics', {})\n\n        video = {\n            \"id\": item.get('id', ''),\n            \"title\": snippet.get('title', ''),\n            \"description\": snippet.get('description', ''),\n            \"publishedAt\": snippet.get('publishedAt'),\n            \"channelId\": snippet.get('channelId', ''),\n            \"channelTitle\": snippet.get('channelTitle', ''),\n            \"videoCategoryId\": snippet.get('categoryId', 0),\n            \"tags\": snippet.get('tags', []),\n            \"videoDuration\": content_details.get('duration', ''),\n            \"videoDefinition\": content_details.get('definition', ''),\n            \"initialViewCount\": statistics.get('viewCount', '0'),\n            \"initialLikeCount\": statistics.get('likeCount', '0'),\n            \"initialFavoriteCount\": statistics.get('favoriteCount', '0'),\n            \"initialCommentCount\": statistics.get('commentCount', '0'),\n            \"collectionDate\": search_date\n        }\n        videos.append(video)\n\n    # return list\n    return {\n        'statusCode': 200,\n        'body': json.dumps(videos)\n    }
\n

Replace the code in Lambda function with the above.

\n

Head to 'Environment Variables' in the 'Configuration' section and add a new variable to store the API key.

\n

\"aws-yt-19\"

\n

Since AWS Lambda doesn't include googleapicleint by default, we need to run the below command in local python environment to package it into a folder, zip and upload it in the Lambda environment.

\n

We'll use Lambda Layers to package this library separately from our code. Using layers allow the package to be used across multiple Lambda functions as well.\nNote that all packages in a Lambda layer need to reside inside a python folder.

\n
mkdir -p layer/python\npip install google-api-python-client -t layer/python\ncd layer\nzip -r googleapiclient_layer.zip python
\n

Go towards the bottom of the page and add a new Layer. Select 'Create a new layer'.

\n

\"aws-yt-20\"

\n

Add a layer name such as googleapiclient, choose the Python runtime, and upload the zip file created in the above steps.

\n

Back in the Lambda function page, go towards the end and select 'Add a layer' again. Select the Layer we created to attach it to the function.

\n

\"aws-yt-21\"

\n

Now let's create a test event so that the function can be executed for testing.\nHead to 'Test' tab and create a test event with an empty json input.

\n

\n

Back in the 'Code' tab, deploy the code, and run the test.

\n

\n

The output will be displayed in the window.

\n

\"aws-yt-24\"

\n

5. Store the raw data in S3

\n

Head to S3 in the AWS Console and create a new S3 bucket to store data.

\n

\n

In the Python code in the Lambda function, replace the json output with the below.\nWe're using partitions to store the data in S3 to make data reads more efficient. In this instance, we're using the collection_date to partition the\ndata into separate folders. S3 will create folders as collection_date=date so that each file can be easily queried.

\n
    # add new imports\n    import os\n    from io import BytesIO\n    import pyarrow as pa\n    import pyarrow.parquet as pq\n    import boto3\n    from datetime import datetime\n    from zoneinfo import ZoneInfo\n    from googleapiclient.discovery import build\n\n    # ------------------------------------------\n\n    # get column names \n    columns = videos[0].keys()\n\n    # convert list of dicts to list of lists\n    videos_list = {key:[item[key] for item in videos] for key in columns}\n\n    # convert list to pyarrow table\n    videos_tb = pa.table(videos_list)\n\n    # init s3 client\n    s3 = boto3.client('s3')\n\n    # use collection_date to partition\n    bucket_name = '<S3 Bucket name>'\n    parquet_file_key = f'raw/videos/collection_date={search_date}/{search_date}.parquet'\n\n    # write to parquet file in memory\n    parquet_buffer = BytesIO()\n    pq.write_table(videos_tb, parquet_buffer)\n\n    # upload to s3\n    s3.put_object(Bucket=bucket_name, Key=parquet_file_key, Body=parquet_buffer.getvalue())\n    print(f'Parquet file uploaded to s3://{bucket_name}/{parquet_file_key}')
\n

Before running the function, we need to add the pyarrow package into Lambda as well.\nWe could try the previous method we used for googleapiclient, but for pyarrow this will result in errors\nbecause some files are dependent on the OS (Mac, Windows or Linux) it is built on. For the pyarrow package to be usable on Lambda,\nit should be built on a Linux machine. Alternatively, you could make use of a public pre-built layer resource available here.\nMake sure to pick the correct Python version and AWS region and obtain the arn value for the resource.

\n

In the Lambda function, select 'Add a Layer' and use the arn to attach the layer.\nI've also updated the Python version of the Lambda function to match the Python version I picked for pyarrow.

\n

\"26\"

\n

Now, if you deploy changes and test the function, it will throw an error saying that the function has timed out after 3 seconds.\nSince writing parquet files need more time, let's update the timeout value to 1 minute.

\n

\"26-2\"

\n

When you execute the function now, it'll throw an error mentioning that permissions are missing.

\n
An error occurred (AccessDenied) when calling the PutObject operation
\n

To resolve this error, we need to enable the IAM role associated with Lambda function to access the S3 bucket.

\n

Go to the Configuration tab and select Permissions. Click on the role name to open it in IAM console.

\n

\"27\"

\n

Select to edit the existing policy.

\n

\"28\"

\n

Select outside of statement to get the option Add new statement on the left side.

\n

\"29\"

\n

Then select s3 as the service to add, and s3:GetObject and s3:PutObject as the allowed actions.\nIn the resource selection, select the S3 bucket created earlier. This should add a snippet similar to below.

\n
        {\n\t\t\t\"Effect\": \"Allow\",\n\t\t\t\"Action\": [\n\t\t\t\t\"s3:PutObject\",\n\t\t\t\t\"s3:GetObject\"\n\t\t\t],\n\t\t\t\"Resource\": [\n\t\t\t\t\"arn:aws:s3:::<bucket_name>/*\"\n\t\t\t]\n\t\t}
\n

Rerun the Lambda function. If you followed all the steps, the parquet file should be created in the S3 bucket.

\n

\n

Congratulations on reaching this point! You've done a great job!!

\n

6. Enable the job to run daily using Amazon EventBridge

\n

Before setting up the daily schedule for the Lambda function, we need to update the publishedBefore and publishedAfter dates to change dynamically in the first API call.

\n

Assume we want to run the daily job at 12.30am AEDT (UTC +11), which is 1.30pm UTC.\nWe can get the videos published from 1.30pm on previous day to 1.30pm the next day.

\n
    from datetime import datetime, timedelta\n\n    # set datetime for search\n    search_date = (datetime.now(ZoneInfo('Australia/Melbourne'))-timedelta(1)).strftime('%Y-%m-%d')\n    previous_date = (datetime.now(ZoneInfo('Australia/Melbourne'))-timedelta(2)).strftime('%Y-%m-%d')\n    utc_time = 'T13:30:00Z'\n    search_date_utc = search_date + utc_time\n    previous_date_utc = previous_date + utc_time
\n

Once this is updated in the API request, we can set up the daily trigger to execute the function using Amazon EventBridge.

\n

Head to EventBridge console and create a rule.\nClick on 'Continue to create rule'.

\n

\"31\"

\n

Set the cron expression to trigger the Lambda function. I've set it to 1.30pm UTC.

\n

\"32\"

\n

Set the Lambda function as the target.

\n

\"33\"

\n

Head back to the Lambda function to verify that the trigger has been added.

\n

\"34\"

\n
    \n
  1. Read data from S3 to get data from API
    2. \n
\n

Let's create a second Lambda function as getYoutubeStats.\nIn the Lambda function, let's get all the data files stored in S3:

\n
import boto3\n\ndef lambda_handler(event, context):\n\n    # init s3 client\n    s3 = boto3.client('s3')\n\n    bucket_name = '<bucket_name>' \n    directory_prefix = 'raw/videos/'\n\n    # get all files in raw/videos/\n    all_files_response = s3.list_objects_v2(Bucket=bucket_name, Prefix=directory_prefix)\n    print(all_files_response)
\n

To run the process, update the IAM role related with this new Lambda function with the permissions to access S3.

\n
        {\n\t\t\t\"Effect\": \"Allow\",\n\t\t\t\"Action\": [\n\t\t\t\t\"s3:GetObject\",\n\t\t\t\t\"s3:PutObject\"\n\t\t\t],\n\t\t\t\"Resource\": [\n\t\t\t\t\"arn:aws:s3:::<bucket_name>/*\"\n\t\t\t]\n\t\t},\n\t\t{\n\t\t\t\"Effect\": \"Allow\",\n\t\t\t\"Action\": [\n\t\t\t\t\"s3:ListBucket\"\n\t\t\t],\n\t\t\t\"Resource\": [\n\t\t\t\t\"arn:aws:s3:::<bucket_name>\"\n\t\t\t]\n\t\t}
\n

When the function is run, the response will have a Contents field, which contains the file names stored in S3. We can use this to get the data in the file.

\n

The completed function is as below:

\n
import os\nfrom io import BytesIO\nimport pyarrow as pa\nimport pyarrow.parquet as pq\nimport boto3\nfrom datetime import datetime, timedelta\nfrom zoneinfo import ZoneInfo\nfrom googleapiclient.discovery import build\n\n\ndef lambda_handler(event, context):\n    # get the API key from Lambda environment variables\n    youtube_api_key = os.environ['YOUTUBE_API_KEY']\n\n    # define api variables\n    api_name = 'youtube'\n    api_version = 'v3'\n\n    # initialise the API client\n    youtube = build(api_name, api_version, developerKey=youtube_api_key)\n\n    # init s3 client\n    s3 = boto3.client('s3')\n\n    # set today's datetime for search\n    search_date = (datetime.now(ZoneInfo('Australia/Melbourne')) - timedelta(1)).strftime('%Y-%m-%d')\n    # get last 6 days for which data should be retrieved\n    today = datetime.now(ZoneInfo('Australia/Melbourne'))\n    search_date_list = [(today-timedelta(i)).strftime('%Y-%m-%d') for i in range(2,8)]\n\n    # get the oldest date for which video data should be retrieved\n    # convert string to datetime\n    oldest_collection_date = datetime.strptime((today - timedelta(7)).strftime('%Y-%m-%d'), '%Y-%m-%d')\n\n    # get data for last 7 days\n    bucket_name = '<bucket_name>'\n    videos = []\n\n    for day in search_date_list:\n        directory_prefix = 'raw/videos'\n        partition_prefix = f'{directory_prefix}/collection_date={day}'\n\n        # check if partition exists in S3\n        file_response = s3.list_objects_v2(Bucket=bucket_name, Prefix=partition_prefix)\n\n        # break loop if latest partition is not available - means previous partitions aren't available as well\n        if 'Contents' not in file_response:\n            break\n        else:\n            # file path\n            file_key = file_response.get('Contents')[0].get('Key')\n            # file date in datetime format\n            file_date = datetime.strptime(day, '%Y-%m-%d') \n            search_date_dt = datetime.strptime(search_date, '%Y-%m-%d')\n            # get object\n            file_obj = s3.get_object(Bucket=bucket_name, Key=file_key)\n            file_table = pq.read_table(BytesIO(file_obj['Body'].read()))\n            video_ids = file_table.column('id').to_pylist()\n\n            # get data for each video\n            video_data_response = youtube.videos().list(\n                part='statistics',\n                id=','.join(video_ids)\n            ).execute()\n\n            for item in video_data_response.get('items', []):\n                statistics = item.get('statistics', {})\n\n                video = {\n                    \"id\": item.get('id', ''),\n                    \"initialCollectionDate\": file_date,\n                    \"collectionDate\": search_date_dt,\n                    \"collectionCount\": (file_date - oldest_collection_date).days + 2,\n                    \"viewCount\": statistics.get('viewCount', '0'),\n                    \"likeCount\": statistics.get('likeCount', '0'),\n                    \"favoriteCount\": statistics.get('favoriteCount', '0'),\n                    \"commentCount\": statistics.get('commentCount', '0')\n                }\n                videos.append(video)\n\n    # save data in s3 if data is available\n    if len(videos) > 0:\n        # get column names\n        columns = videos[0].keys()\n\n        # convert list of dicts to list of lists\n        videos_list = {key: [item[key] for item in videos] for key in columns}\n\n        # convert list to pyarrow table\n        videos_tb = pa.table(videos_list)\n\n        # write to parquet file in memory\n        parquet_buffer = BytesIO()\n        pq.write_table(videos_tb, parquet_buffer)\n\n        # upload to s3\n        parquet_file_key = f'raw/video_stats/collection_date={search_date}/{search_date}.parquet'\n        s3.put_object(Bucket=bucket_name, Key=parquet_file_key, Body=parquet_buffer.getvalue())\n        print(f'Parquet file uploaded to s3://{bucket_name}/{parquet_file_key}')\n    else:\n        print(f'No data available before {search_date_list[0]}')
\n

Before executing the function, make sure you have added environment variables, and attached the required layers similar to what we did for 'getYoutubeData' function.

\n

Finally, add the new function to be triggered with the same EventBridge rule.

\n

\"35\"

\n

Now we have setup the data extract Lambda functions to get data using the YouTube API!

\n

Congratulations on reaching this far!

\n

Read the next steps of the project - using AWS Glue to transform data and store in Redshift - here in Part 2.

","frontmatter":{"date":"December 11, 2024","path":"/get-youtube-data-aws","title":"Extract and load YouTube data in S3 using AWS Lambda","tags":["Project","AWS","ETL","AWS Lambda"]}}},"pageContext":{}},"staticQueryHashes":["3649515864"]}