{"componentChunkName":"component---src-templates-blog-post-js","path":"/generate-your-own-dataset","result":{"data":{"markdownRemark":{"html":"

\"data-gen1\"

\n
Photo by Tech Daily on Unsplash
\n

Hey there!

\n

Have you ever had the idea for a personal project but couldn't find the right dataset for you?

\n

I was in your shoes when I wanted to create a Power BI report for HR (Human Resource) data with a monthly breakdown.\n
\nAnd update it every month with a new data file, while keeping the old data files in place.

\n

To get started, I needed a dataset with employee data for several months, separately for each month.

\n

Fortunately, I came across the HR dataset created by obviEnce, which is also the\ndataset used in the Microsoft Power BI HR sample.\nIt has employee data for each month in a separate csv file, which I could use to build a number of HR\ndata metrics like headcount and number of new hires.

\n

However, to make the project more interesting, I thought of creating another set of data files to replicate a\nsecond data source. This dataset would contain data from the hiring process and help create other metrics such as the number of\nvacancies and filled roles in the company over time.

\n

\"data-gen2\"

\n
The Project Idea
\n

In this post, I'm going to go through the process of how I used Python to generate the vacancy dataset. This may be useful if you\never wanted to build something similar and did not have the access to an open dataset with multiple files.

\n

Before we get started,

\n

It may be useful to look at pros and cons in this approach. If you're able to find one, it's always better to use a dataset\nfrom real world over a generated dataset. It brings in the 'real-world' aspect, because it'll have actual patterns\nthat you can use to build a story around your data.

\n

However, because my main goal for this project is simply visualising\ndata using Power BI, it's sufficient to use a generated dataset. I tried to bring in the real-world aspect by using\nrandom sampling, a hiring timeline and predefined proportions to add variety to the data. I'll explain how to\nincorporate these ideas in the next sections of this post.

\n

A brief look at the first data source - Employee data

\n

Before we dive into vacancy data, let's first look at the data from obviEnce. Because both data sources should be\ncoming from the same company, the vacancy dataset will need to share some information from this data.

\n

These are the columns from our first data source:

\n

\"data-gen3\"

\n
Data Description for the first dataset
\n

From these variables, I've chosen to use the 'FP' and 'BU Region' columns to indicate vacancies for\nfull/part-time positions in different BU regions.

\n

Let's also have a look at the count of new hires (those who started working, not just hired) from the Employee dataset.

\n

In my Python notebook, I first import the packages.

\n
import pandas as pd\nimport numpy as np\nimport random\nimport calendar\nimport glob\nimport os\n\nimport matplotlib.pyplot as plt\n%matplotlib inline
\n

And plot the number of new hires:

\n
# plot new hire count in employee data\n\n# get current working directory\ncwd = os.path.abspath(os.getcwd())\n# get all files in directory\nfiles = glob.glob(cwd + \"/Employee data/*\")\n\nmonths = []\nnewHires = []\n\nfor file in files:\n    df = pd.read_csv(file)\n    # get the count of new hires\n    newHires.append(df['isNewHire'].sum())\n    # get the month for the file\n    months.append(pd.to_datetime(df['date'][0], dayfirst=True).month)\n                 \n# sort new hires based on month\nnewHires = [x for _,x in sorted(zip(months, newHires))]\nmonths = sorted(months)\n# replace month numbers with names\nmonths = [calendar.month_name[i] for i in months]\n\n# plot new hires\nfig, ax = plt.subplots(figsize=(15,8))\nbars = ax.bar(months, newHires)\n# add data labels\nfor i,v in enumerate(newHires):\n    ax.text(i,v+20, int(v), ha='center')\nplt.title('New Hires')\nplt.show()\n\nprint('All new starters for the year: ', sum(newHires))
\n

which gives us the graph below:

\n

\"data-gen31\"

\n
New hire count from the first dataset
\n

As you can see, there is some variation in the total of new hires over the year, and the total new hire count is at 19073.\nTo keep the counts similar, we'll create 20000 vacancies in our vacancy dataset.

\n

Timeframe for the dataset

\n

Next, let's look at the time duration for which we will be generating data. As I'll be using the data within January\n2014 to December 2014 from the first data source, I've defined the same time period for vacancy dataset as well.

\n

Let's define the number of vacancies for the year, and start and end dates for the timeframe.

\n
n_vacancies = 20000\nyear = '2014'\nmin_date = pd.to_datetime(year + '/01/01')\nmax_date = pd.to_datetime(year + '/12/31')
\n

Data Generation

\n

I'll be following a two-step process in data generation. First, I created the complete list of vacancies for the timeframe\nof one year.

\n

In the next step, I'll be slicing and storing the data in separate files for each month.

\n

\"data-gen4\"

\n
Overview of the Data Generation Process
\n

1. Generate all vacancy data

\n

Generate IDs and add categories from first dataset

\n

We'll get started by creating random and unique IDs for the vacancies.\n
And randomly assign categories from FP and BU Region columns from the first dataset.

\n

\"data-gen5\"

\n
First three columns in the vacancy dataset
\n

In my function to generate all vacancy data, I'm adding the ID, FP and BU Region columns.\nTo generate IDs, I'm using random.sample() function to do random sampling without replacement. To assign FP and BU Region,\nI'm using random.choice() from Numpy to pick items with replacement.

\n
# generate all vacancy data\n\ndef generate_vacancy_data(n_vacancies):\n    vacancy_df = pd.DataFrame()\n    # create random ids for each row\n    vacancy_df['ID'] = random.sample(range(1,n_vacancies+1), n_vacancies)\n    # randomly add values for FP and BU Region columns\n    vacancy_df['FP'] = np.random.choice(['F','P'], n_vacancies, replace=True)\n    regions = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, \n                              19, 20, 21, 22, 23, 24, 94, 95, 96, 97, 98, 99]\n    vacancy_df['BU Region'] = np.random.choice(regions, n_vacancies, replace=True)
\n

Generate dates based on the hiring timeline

\n

Next, I'll be adding more columns to explain the status of vacancies over time.
\nFor this purpose, I've defined a hypothetical hiring timeline that we can assume that the company is using in their hiring process.\nThe image below shows the major milestones for a single vacancy.

\n

\"data-gen6\"

\n
Hiring timeline
\n

I had fun creating this using my own imagination. The main purpose was to bring in some aspects followed in a\nreal-world hiring process.

\n

Each vacancy would have an Approved Date which indicates when the vacancy was created. To generate random approved dates\nspread across the year, I'm using pd.to_timedelta() function to add a random number of days to the first date of the year.

\n
    # get the number of days in the time period\n    n_days = (max_date - min_date).days + 1\n    # generate random approved dates\n    vacancy_df['Approved'] = min_date + pd.to_timedelta(np.random.randint(n_days, size=n_vacancies), unit='d')
\n

Usually companies have some of their vacancies put on hold. To replicate this, a defined portion of 1% of all\nvacancies were to be put on hold. The On hold date would fall within 10-30 days from the Approved date.

\n
    # add empty column filled with NaT\n    vacancy_df['On hold'] = pd.NaT\n    # generate 1% on hold vacancies by selecting 1% of rows\n    a = vacancy_df.sample(frac=0.01)\n    # generate random on hold dates for the selected rows in 10-30 days after approved date\n    a['On hold'] = a['Approved'] + pd.to_timedelta(np.random.randint(low=10, high=30, size=a.shape[0]), unit='d')\n    # replace modified rows in original dataset\n    vacancy_df.loc[a.index, 'On hold'] = a['On hold'].dt.date
\n

Then, I'm using the same approach to add the Sourcing start, Interview start, Interview end, Offered and Filled\ndates based on the hiring timeline. Finally, I've removed the date values for On hold vacancies.

\n

\"data-gen7\"

\n
Generated columns in the Vacancy dataset
\n

With the addition of this section, we now have the complete function to generate all\nvacancy data.

\n
# generate all vacancy data\n\ndef generate_vacancy_data(n_vacancies):\n    vacancy_df = pd.DataFrame()\n    # create random ids for each row\n    vacancy_df['ID'] = random.sample(range(1,n_vacancies+1), n_vacancies)\n    # randomly add values for FP and BU Region columns\n    vacancy_df['FP'] = np.random.choice(['F','P'], n_vacancies, replace=True)\n    regions = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, \n                              19, 20, 21, 22, 23, 24, 94, 95, 96, 97, 98, 99]\n    vacancy_df['BU Region'] = np.random.choice(regions, n_vacancies, replace=True)\n    \n    # get the number of days in the time period\n    n_days = (max_date - min_date).days + 1\n    # generate random approved dates\n    vacancy_df['Approved'] = min_date + pd.to_timedelta(np.random.randint(n_days, size=n_vacancies), unit='d')\n    # add empty column filled with NaT\n    vacancy_df['On hold'] = pd.NaT\n    # generate 1% on hold vacancies by selecting 1% of rows\n    a = vacancy_df.sample(frac=0.01)\n    # generate random on hold dates for the selected rows in 10-30 days after approved date\n    a['On hold'] = a['Approved'] + pd.to_timedelta(np.random.randint(low=10, high=30, size=a.shape[0]), unit='d')\n    # replace modified rows in original dataset\n    vacancy_df.loc[a.index, 'On hold'] = a['On hold'].dt.date\n    \n    # generate sourcing start date to be within 5-10 days after approved date\n    vacancy_df['Sourcing start'] = vacancy_df['Approved'] + pd.to_timedelta(np.random.randint(low=5, high=10, size=n_vacancies), unit='d')\n    # generate Interview start date to be within 10-20 days after sourcing start date\n    vacancy_df['Interview start'] = vacancy_df['Sourcing start'] + pd.to_timedelta(np.random.randint(low=10, high=20, size=n_vacancies), unit='d')\n    # generate Interview end date to be within 15-30 days after interview start date\n    vacancy_df['Interview end'] = vacancy_df['Interview start'] + pd.to_timedelta(np.random.randint(low=15, high=30, size=n_vacancies), unit='d')\n    # generate Offered date to be within 5-10 days after interview end date\n    vacancy_df['Offered'] = vacancy_df['Interview end'] + pd.to_timedelta(np.random.randint(low=5, high=10, size=n_vacancies), unit='d')\n    # generate Filled date to be within 5-10 days after offered date\n    vacancy_df['Filled'] = vacancy_df['Offered'] + pd.to_timedelta(np.random.randint(low=5, high=10, size=n_vacancies), unit='d')\n\n    # remove values for on hold vacancies\n    vacancy_df.loc[a.index, ['Sourcing start', 'Interview start', 'Interview end', 'Offered', 'Filled']] = pd.NaT\n\n    return vacancy_df
\n

I'm sure you're curious to see how it would turn out after running this function.

\n

And we've got the answer here:

\n
# generate complete dataset\nvacancy_df = generate_vacancy_data(n_vacancies)\nvacancy_df
\n

\"data-gen8\"

\n
Vacancy data generation output
\n

Why I'm getting a different output?

\n

If you tried this out, you may notice that you get a different output from what's shown here. This is because we use random\nnumbers to generate the data. If you prefer to get the same output every time, set the seed for random numbers before\nexecuting the above function. This needs to be done for both Python and NumPy random number generators.

\n

This post is a helpful guide to using random seed in NumPy.

\n
# set seed for Python random generator\nrandom.seed(42)\n\n# create NumPy random number generator\nrand_gen = np.random.RandomState(42)\n\n# replace np.random with rand_gen
\n

2. Split data into separate files

\n

Now that we have the complete vacancy data for the year, we can move to the next step and separate the data for each month.

\n

For each month, I'm going to filter the dates less than or equal to the last date for the month, and replace other\ndate values with NaT.

\n

Next, I'll remove the roles which have Filled dates in previous months.

\n

Finally, I'll be adding a new column called Status to indicate the status for each vacancy at the end of the month.

\n

I'll create a new function for this purpose, and run it for each month in our time period.

\n
# create data files for each month\n\ndef create_monthly_df(month):\n    # get start date for the month\n    month_start = pd.to_datetime(year + '/'+str(month)+'/01')\n    # get end date for the month\n    month_end = month_start + pd.to_timedelta(calendar.monthrange(int(year), month)[1]-1, unit='d')\n    \n    # create monthly data\n    monthly_df = vacancy_df.copy()\n    \n    # replace dates after month end date with NaT\n    monthly_df[monthly_df.columns[3:]] = monthly_df.iloc[:,3:].where(monthly_df.iloc[:,3:]<=month_end, pd.NaT)\n    \n    # replace rows with NaT if Filled date is in a previous month\n    monthly_df[monthly_df.columns[3:]] = monthly_df.iloc[:,3:].where(\n        (pd.isnull(monthly_df.iloc[:,-1])) | (monthly_df.iloc[:,-1]>=month_start), pd.NaT)\n    \n    # remove rows with all blank values using index\n    empty_index = monthly_df[monthly_df.iloc[:,3:].isnull().all(axis=1)].index\n    monthly_df = monthly_df.drop(index=empty_index).reset_index()\n\n    # convert On hold column to datetime to be able to compare in the next step with idxmax\n    monthly_df['On hold'] = pd.to_datetime(monthly_df['On hold'], errors='coerce')\n    # add status column by using the max date value from the specified columns\n    monthly_df['Status'] = monthly_df[['Approved', 'On hold', 'Sourcing start', \n                                       'Interview start', 'Interview end', 'Offered', 'Filled']].idxmax(axis=1)\n    # remove previous index column \n    monthly_df = monthly_df.drop(['index'], axis=1)\n    \n    # remove NaT values\n    monthly_df = monthly_df.fillna('')\n    \n    # write to csv\n    file_name = 'Vacancy data/' +calendar.month_name[month] + '-' + year + '.csv'\n    monthly_df.to_csv(file_name, index=False)\n\n\n# generate monthly files\nfor i in range(12):\n    create_monthly_df(i+1)
\n

As you can see, we have a new Status column in the output for 2014 April.

\n

\"data-gen9\"

\n
First six rows of the final vacancy dataset output for 2014 April
\n

If you tried this out, you will have 12 csv files created for vacancy data in each month.\n
Each file shows the status of the vacancies at the end of the month.

\n

\"data-gen10\"

\n
Generated data files
\n

I had a brief look at the count of filled roles over the year using the generated data.

\n
# plot filled roles in vacancy data\n\n# get all files in directory\nvac_files = glob.glob(cwd + \"/Vacancy data/*\")\n\nmonths = []\nfilledRoles = []\n\nfor file in vac_files:\n    df = pd.read_csv(file)\n    # get the count of filled roles\n    filledRoles.append(df[df['Status']=='Filled']['ID'].count())\n    # get the month for the file\n    months.append(pd.to_datetime('1-'+file.split('/')[-1].split('.')[0], dayfirst=True).month)\n                 \n# sort filled roles based on month\nfilledRoles = [x for _,x in sorted(zip(months, filledRoles))]\nmonths = sorted(months)\n# replace month numbers with names\nmonths = [calendar.month_name[i] for i in months]\n\n# plot new hires\nfig, ax = plt.subplots(figsize=(15,8))\nbars = ax.bar(months, filledRoles)\n# add data labels\nfor i,v in enumerate(filledRoles):\n    ax.text(i,v+20, int(v), ha='center')\nplt.title('Filled Roles')\nplt.show()\n\nprint('All filled roles for the year: ', sum(filledRoles))
\n

The output shows that we have a fairly similar distribution over the year as I used random sampling to generate the data.

\n

\"data-gen11\"

\n
Total number of filled roles using generated data
\n

Although the distribution is different, we can use the generated data to explore more aspects related to the hiring\nprocess - such as time taken to source candidates and time taken to fill roles.

\n

You may be wondering about the different numbers for total hires from the first dataset, and the total filled\nroles from generated data. Usually in companies, these two values tend to be different because although a role\nwas filled in one month, the employee may start working on a later date.

\n

And that's it! That's how I generated my own dataset so that I can use it for my personal project.

\n

Check my GitHub repository to see the complete code for\nthis project, and view the generated dataset in Kaggle.

\n

Have fun with Python!

","frontmatter":{"date":"November 18, 2022","path":"/generate-your-own-dataset","title":"Build your own dataset for personal projects","tags":["Datasets","Project","Python","Data Visualisation"]}}},"pageContext":{}},"staticQueryHashes":["3649515864"]}