{"id":17608,"date":"2023-03-06T09:00:47","date_gmt":"2023-03-06T14:00:47","guid":{"rendered":"https:\/\/qxf2.com\/blog\/?p=17608"},"modified":"2023-03-06T09:00:47","modified_gmt":"2023-03-06T14:00:47","slug":"setting-up-synthetic-data-in-neo4j","status":"publish","type":"post","link":"https:\/\/qxf2.com\/blog\/setting-up-synthetic-data-in-neo4j\/","title":{"rendered":"Setting up Synthetic data in Neo4j"},"content":{"rendered":"

Qxf2<\/a> engineers are fans of using synthetic data for testing. We have used this technique for years now. But until recently, our experience was limited to SQL. We needed to work with neo4j in one of our projects. There was enough custom Python code and Cypher queries that we had to write as part of creating, backing up and restoring data. So we thought of sharing some of it here. <\/p>\n

PS: To help you play along, we are going to implement some seed data for one of our internal projects. The code is open sourced and you can find “Further Reading” section. <\/p>\n

<\/p>\n

These are some principles we use when we designed synthetic data:<\/p>\n

1. Understand the Data Model:<\/strong> It is important to have a clear understanding of the data model before creating synthetic data. This allows for the creation of relevant and meaningful synthetic data, which can help identify potential issues.<\/p>\n

2. Easy Interpretation:<\/strong> Synthetic data should be designed in such a way that its purpose can be easily grasped by a viewer at a glance. This implies that the data must be easy to comprehend and interpret.<\/p>\n

3. Vary the Data:<\/strong> It is important to vary the synthetic data to test different scenarios. This includes varying the data types, values, and relationships to ensure that the database can handle a wide range of data.<\/p>\n

4. Make the data extensible:<\/strong> When we create synthetic data for the first time in a project, it is important to design it in such a way that the data is easily extensible. We want to be able to add new data in future without affecting any of the existing tests. <\/p>\n

<\/p>\n

Understanding the data model<\/h3>\n

Before creating the synthetic data, it’s important to have a clear understanding of the data model. For this example, we’ll be creating synthetic data for the Qxf2-survey<\/a> app, which uses the Neo4j database. The following ER diagram represents the data model we’ll be working with:
\n

\"Neo4j<\/a>
Entity Relationship Diagram of Neo4j Database model for Qxf2-Survey App<\/figcaption><\/figure><\/p>\n

1. As you can see from the above diagram, The Neo4j data model consists of two node labels: “Employees<\/strong>” and “Technology<\/strong>“. The “Employees<\/strong>” label represents individuals in the organization and has properties such as author_name<\/code>, firstName<\/code>, lastName<\/code>, fullName<\/code>, email<\/code>, ID<\/code>, and status<\/code>. The “Technology<\/strong>” label represents the various technologies used in the organization and has properties such as technology_name<\/code> and first_seen<\/code>, both of which are stored as date strings.<\/p>\n

2. Also, we see that there is a relationship between the “Employee<\/strong>” and “Technology<\/strong>” nodes called “KNOWS<\/strong>“, which denotes the knowledge that an employee has about a technology. The “KNOWS” relationship has one property named “learnt_dates<\/code>“, which stores the dates on which the employee learned about the technology.<\/p>\n

3. In addition to the “KNOWS<\/strong>” relationship, there are two relationships among the “Employees<\/strong>” nodes: “Given<\/strong>” and “Taken<\/strong>“. The “Given<\/strong>” relationship represents the help given by an employee to another and has a property named “helpgiven<\/code>“, which stores the date on which the help was given. The “Taken<\/strong>” relationship represents the help taken by an employee from another and has a property named “helptaken<\/code>“, which stores the date on which the help was taken.<\/p>\n

Understanding the API<\/h3>\n

1. In this example, our focus is to create synthetic data to test a single API endpoint from the Qxf2-survey<\/a><\/strong> app. We will be testing the endpoint \/survey\/admin\/QElo_filter_response<\/code>, which is designed to return the help responses that occurred within a specific date range.<\/p>\n

2. The endpoint queries the GIVEN<\/strong> and TAKEN<\/strong> relationship amongst the employees, filters the helpgiven<\/code> and helptaken<\/code> dates based on the date parameters passed by the user and returns a response similar to this:<\/p>\n

\r\n[\r\n  {\r\n    \"respondent_id\": 1,\r\n    \"date\": \"2022-02-18\",\r\n    \"question_no\": 1,\r\n    \"answer\": \"dummy_user\"\r\n  },\r\n  {\r\n    \"respondent_id\": 2,\r\n    \"date\": \"2022-02-18\",\r\n    \"question_no\": 2,\r\n    \"answer\": \"dummy_user\"\r\n  }\r\n...\r\n...\r\n]\r\n<\/pre>\n
3. In the above response, The “respondent_id<\/code>” field identifies the employee who submitted the survey response, while the “date<\/code>” field indicates the date on which the employee either gave or received help from another employee. The “question_no<\/code>” field provides additional detail on the type of help exchanged, with a value of 1 indicating “TAKEN<\/strong>” help and a value of 2 indicating “GIVEN<\/strong>” help. Finally, the “answer<\/code>” field allows you to identify the employee who provided or received help from the respondent.<\/p>\n
 Creating the synthetic data for Neo4j<\/h3>\n
Now that we have a clear understanding of the data model and the API to test, we can start creating our synthetic data.
\nWe will break down the process to create the synthetic data into two parts:
\nA. Creating the synthetic Nodes
\nB. Creating the synthetic relationships<\/p>\n
A. Creating the synthetic Nodes<\/h4>\n
As we know, the API endpoint we are testing queries the ‘GIVEN<\/strong>‘ and ‘TAKEN<\/strong>‘ relationships. In addition, we need to ensure that it does not return help responses from inactive users as well. To test this, we will create three separate nodes: one for an employee who ‘Gives’ help, another for an employee who ‘Takes’ help, and a third node for an inactive employee. This will allow us to test the endpoint accurately and effectively to ensure that it works as intended.<\/p>\n
1. First, open your Neo4j Browser or desktop application, create a new database ‘neo4j’ where you want to create the nodes and start the database.<\/p>\n
2. In the Cypher editor, lets type the following command to create our first node:<\/p>\n
\r\nCREATE (:Employees {\r\n  author_name: \"Generous Giver\",\r\n  lastName: \"Giver\",\r\n  firstName: \"Generous\",\r\n  fullName: \"Generous Giver\",\r\n  ID: 1,\r\n  email: \"generousgiver@qxf2.com\",\r\n  status: \"Y\"\r\n})\r\n<\/pre>\n
3. Pay attention to the naming, we have named our employee as Generous Giver<\/em><\/strong>. It is understood by common knowledge that the node represents an employee who gives a lot of help to other employees. Our synthetic data should be commonly understood  and meaningful. Therefore, we should strive to use similar naming conventions for all our synthetic data to ensure consistency and clarity<\/p>\n
4. Now, press the “Run” button to execute the command. You should see a message indicating that one node has been created.
\n
\"Neo4j<\/a>
Successfully creating a new node in Neo4j<\/figcaption><\/figure><\/p>\n
5. Now that we have our first node created, let’s Repeat the process to create two other nodes:<\/p>\n
\r\nCREATE (:Employees {\r\n  author_name: \"Generous Taker\",\r\n  lastName: \"Taker\",\r\n  firstName: \"Generous\",\r\n  fullName: \"Generous Taker\",\r\n  ID: 2,\r\n  email: \"generoustaker@qxf2.com\",\r\n  status: \"Y\"\r\n})\r\n\r\nCREATE (:Employees {\r\n  author_name: \"Inactive User\",\r\n  lastName: \"User\",\r\n  firstName: \"Inactive\",\r\n  fullName: \"Inactive User\",\r\n  ID: 3,\r\n  email: \"inactive_user@qxf2.com\",\r\n  status: \"N\"\r\n})\r\n\r\n<\/pre>\n
6. We have named the two employees as Generous Taker<\/em><\/strong> and Inactive User<\/em><\/strong>. Again as the name implies , it is understood that Generous Taker<\/em><\/strong> represents an employee who receives a lot of help, while Inactive User<\/em><\/strong> is a an employee who is no longer active with the company.<\/p>\n
Now that we have our nodes created, let’s see how can create the relationships for our synthetic data<\/p>\n
 Creating the synthetic relationships<\/h4>\n
From the data model shown above, we know that there can be two relationships ‘GIVEN<\/strong>‘ or ‘TAKEN<\/strong>‘ amongst  Employees<\/strong>. So lets add these relations to our synthetic data.<\/p>\n
1. First, lets add a relationship ‘GIVEN<\/strong>‘ between employees Generous Giver<\/em><\/strong> and Generous Taker<\/em> <\/strong>, set the helpgiven<\/strong> property and assign some dates to it.<\/p>\n
\r\nMATCH (a:Employees) WHERE a.lastName=\"Giver\" \r\nMATCH (b:Employees) WHERE b.lastName=\"Taker\"\r\nMERGE (a)-[R:GIVEN]->(b)\r\nSET R.helpgiven = [\"1970-01-02\", \"1970-01-09\", \"1970-01-16\"]\r\n<\/pre>\n
If you notice, we have assigned some past dates to the helpgiven<\/strong> property, which corresponds to the beginning of the Unix epoch. This is done to separate the synthetic data from real data and make it easier for a person to distinguish between them. You can use similar techniques or patterns for your use case as well.<\/p>\n
2. Similarly, lets create a TAKEN<\/strong> relationship between employees Generous Taker<\/em><\/strong> and Generous Giver<\/em><\/strong>, set the helptaken<\/strong> relationship property, and assign some dates to it<\/p>\n
\r\nMATCH (a:Employees) WHERE a.lastName=\"Taker\" \r\nMATCH (b:Employees) WHERE b.lastName=\"Giver\"\r\nMERGE (a)-[R:TAKEN]->(b)\r\nSET R.helptaken = [\"1970-01-02\", \"1970-01-09\", \"1970-01-16\"]\r\n<\/pre>\n
3. Since we also need to verify the API response for inactive users as well, lets create a ‘GIVEN<\/strong>‘ relationship between Generous Giver<\/em><\/strong> and Inactive User<\/em><\/strong> <\/p>\n
\r\nMATCH (a:Employees) WHERE a.fullName=\"Generous Giver\" \r\nMATCH (b:Employees) WHERE b.fullName=\"Inactive User\"\r\nMERGE (a)-[R:GIVEN]->(b)\r\nSET R.helptaken = [\"1975-01-10\"]\r\n<\/pre>\n
4. Let’s also create a ‘GIVEN’ relationship between Inactive User<\/em><\/strong> and Generous Taker<\/em> <\/strong><\/p>\n
\r\nMATCH (a:Employees) WHERE a.fullName=\"Inactive User\" \r\nMATCH (b:Employees) WHERE b.fullName=\"Generous Taker\"\r\nMERGE (a)-[R:GIVEN]->(b)\r\nSET R.helptaken = [\"1975-01-17\"]\r\n<\/pre>\n
5. Now, run the following query:<\/p>\n
\r\nMATCH (n:Employees) \r\nRETURN n\r\n<\/pre>\n
You should see a graph similar to this:
\n
\"Graph<\/a>
Graph representation of synthetic data created for Neo4j<\/figcaption><\/figure><\/p>\n
Finally, we are done setting up our synthetic data. Next lets take a look at how we can backup the synthetic data we created.<\/p>\n
 Creating a backup of the synthetic data in Neo4j<\/h3>\n
We will be using the ‘neo4j-backup’ Python library to create a script that would backup our database. I prefer this approach over the traditional method of creating a dump file for backup and restore, as it can be challenging to automate the latter.<\/p>\n
 Initial setup <\/h4>\n
1. Before starting , you need to have at least Python version 3.6<\/strong> or above installed on your machine.<\/p>\n
2. Let’s start off by installing the necessary python libraries:<\/p>\n
\r\npip install neo4j neo4j_backup decouple\r\n<\/pre>\n
3. Next, let’s create a new Python file and name it “neo4j_backup_script.py<\/em>“. <\/p>\n
We are all set to start writing the script!<\/p>\n
 Writing the script to backup the synthetic data<\/h4>\n
1. Let’s start by importing the libraries that we would be using in our script.<\/p>\n
\r\nfrom neo4j import GraphDatabase\r\nfrom neo4j_backup import Extractor\r\nfrom decouple import config\r\nimport shutil\r\nimport argparse\r\nimport os\r\n<\/pre>\n
2. Now, lets define the variables we need to connect to our Neo4j database in an environment file. Create the .env<\/code> file in the same directory as our python script.<\/p>\n
\r\nDATABASE_HOST=\"bolt:\/\/localhost:7687\"\r\nDATABASE_USERNAME=\"neo4j\"\r\nDATABASE_PASSWORD=\"dummy-password\"\r\n<\/pre>\n
Note:<\/strong> To prevent the credentials from being exposed, it’s important to add the .env<\/code> file to your .gitignore<\/code>. This will ensure that the file is not included in version control.<\/p>\n
3. Next, let’s fetch the environment variables we created in the previous step in our python script. We will be using the config<\/code> function from the decouple<\/code> library to achieve this.<\/p>\n
\r\nHOSTNAME = config(\"DATABASE_HOST\")\r\nUSERNAME = config(\"DATABASE_USERNAME\")\r\nPASSWORD = config(\"DATABASE_PASSWORD\")\r\n<\/pre>\n
4. Now that we have a way to connect to the database, our script also needs to know the name of our database and the directory where the backup should be stored. So let\u2019s use the argparse<\/code> module to get these values from the user as command line arguments.<\/p>\n
\r\nparser = argparse.ArgumentParser()\r\n\r\nparser.add_argument(\"--database_name\", type=str, help=\"Provide the name of the database\", nargs='?', default=\"neo4j\", const=0)\r\nparser.add_argument(\"--save_dir\", type=str, help=\"Provide the name of the directory in which the backup would be stored\", nargs='?', default=\"synthetic_data\", const=0)\r\n\r\nargs = parser.parse_args()\r\ndatabase = args.database_name\r\nproject_dir = args.save_dir\r\n<\/pre>\n
The --database_name<\/code> argument is used to specify the name of the database that needs to be backed up, and the --save_dir<\/code> argument is used to specify the directory where the backup is to be stored.<\/p>\n
5. Next, lets define the connection settings for the Neo4j database and create a driver object using the GraphDatabase.driver()<\/code> function. This will help us connect to our database.<\/p>\n
\r\nencrypted = False\r\ntrust = \"TRUST_ALL_CERTIFICATES\"\r\ndriver = GraphDatabase.driver(HOSTNAME, auth=(USERNAME, PASSWORD), encrypted=encrypted, trust=trust)\r\n<\/pre>\n
Note:<\/strong> The code above sets encrypted=False<\/code> for an unencrypted connection to the Neo4j database. Only use this for local databases. For remote servers, use encrypted=True<\/code> for encrypted SSL\/TLS connections to secure transmitted data.<\/p>\n
6. Now that we have connected to our database, we can finally extract the data to create the backup.  We can do this  using the Extractor class from the neo4j_backup<\/code> library.<\/p>\n
\r\ninput_yes = False\r\ncompress = True\r\nextractor = Extractor(project_dir=project_dir, driver=driver, database=database, input_yes=input_yes, compress=compress)\r\nextractor.extract_data()\r\n<\/pre>\n
7. Finally, let’s compress the backup directory using the make_archive()<\/code> function from the shutil<\/code> library, and then delete the original directory using the rmtree()<\/code> function.<\/p>\n
\r\nshutil.make_archive(project_dir, 'zip', project_dir)\r\nshutil.rmtree(project_dir)\r\n<\/pre>\n
8. Our complete script should look like this:<\/p>\n
\r\n\"\"\"\r\nBackup Neo4j database\r\n\"\"\"\r\nfrom neo4j import GraphDatabase\r\nfrom neo4j_backup import Extractor\r\nfrom decouple import config\r\nimport shutil\r\nimport argparse\r\nimport os\r\n\r\n# Grabbing environment variables\r\nHOSTNAME = config(\"DATABASE_HOST\")\r\nUSERNAME = config(\"DATABASE_USERNAME\")\r\nPASSWORD = config(\"DATABASE_PASSWORD\")\r\n\r\nif __name__ == \"__main__\":\r\n\r\n    #Add command line arguments to fetch import file and database name\r\n    parser = argparse.ArgumentParser()\r\n\r\n    #Command line argument to fetch database name. Database name is taken as 'neo4j' if no argument is specified\r\n    parser.add_argument(\"--database_name\", type=str, help=\"Provide the name of the database\",\r\n                        nargs='?', default=\"neo4j\", const=0)\r\n    parser.add_argument(\"--save_dir\", type=str, help=\"Provide the name of the directory in which the backup would be stored\",\r\n                        nargs='?', default=\"synthetic_data\", const=0)\r\n    args = parser.parse_args()\r\n    database = args.database_name\r\n    encrypted = False\r\n    trust = \"TRUST_ALL_CERTIFICATES\"\r\n    driver = GraphDatabase.driver(HOSTNAME, auth=(USERNAME, PASSWORD), encrypted=encrypted, trust=trust)\r\n    project_dir = args.save_dir\r\n    input_yes = False\r\n    compress = True\r\n\r\n    #Extract data from database and store the backup\r\n    extractor = Extractor(project_dir=project_dir, driver=driver, database=database,\r\n                          input_yes=input_yes, compress=compress)\r\n    extractor.extract_data()\r\n    shutil.make_archive(project_dir, 'zip', project_dir)\r\n    shutil.rmtree(project_dir)\r\n\r\n<\/pre>\n
 Running the backup script<\/h4>\n
1. To run the backup script, first make sure that the neo4j database you want to backup is running.<\/p>\n
2. Open a terminal, navigate to the location of the neo4j_backup_script.py<\/em> file and run the following command:<\/p>\n
\r\npython neo4j_backup_script.py --database_name  --save_dir \r\n<\/pre>\n
Make sure to replace database_name<\/code> with the name of the database you want to backup, and save_directory<\/code>with the directory where you want to save the backup file.
\n
\"Execution<\/a>
Executing the script to backup the Neo4j database<\/figcaption><\/figure><\/p>\n
3. Once the script finish running, navigate to the save_dir<\/code> location that you specified in the command to run the script. You should see a new zip file created which contains the backup of your database.<\/p>\n
Kudos! We have successfully backed up our synthetic data. Next, let’s have a look at how we can restore the data that we backed up.<\/p>\n
 Restoring the synthetic data <\/h3>\n
In the previous section, we used the neo4j-backup library to create a backup of our database. We will be using same library again to restore our synthetic data as well.
\nWe will not cover the initial setup required for this script, as it is identical to the steps outlined in the previous section to create the backup script.<\/p>\n
 Writing the script to restore the synthetic data in Neo4j<\/h4>\n
Let’s start by creating a new python file and name it “neo4j_restore_script.py”<\/em>.
\nWith the file in place, we’re all set to start writing the script to restore the synthetic data.
\nThe creation of the script will be divided into the following sections:
\nA. Extracting the backup file
\nB. Importing the backup data
\nC. Putting it all together<\/p>\n
 A. Extracting the backup file<\/h5>\n
In this section, we’ll explore the process of extracting the backup file and preparing it for import.
\n1. First lets import the libraries that we would be using in our script.<\/p>\n
\r\nimport sys\r\nfrom neo4j import GraphDatabase\r\nfrom neo4j_backup import Importer\r\nfrom py2neo import Graph\r\nfrom decouple import config\r\nfrom zipfile import ZipFile\r\nimport argparse\r\n<\/pre>\n
2. Next, in a similar fashion to the backup script, we must define the necessary variables for connecting to our database. As we’re using the same variables as in the backup script, there’s no need to define or add any new variables to our environment file. Therefore, we can simply retrieve these environment variables in our script.<\/p>\n
\r\n# Grabbing environment variables\r\nHOSTNAME = config(\"DATABASE_HOST\")\r\nUSERNAME = config(\"DATABASE_USERNAME\")\r\nPASSWORD = config(\"DATABASE_PASSWORD\")\r\n<\/pre>\n
3. Now, lets go ahead and define the main<\/code> function. As with the backup script, we’ll use the argparse<\/code> module to define the command line arguments that the restore script will accept. To be precise, we’ll specify two arguments database_name<\/code> and import_file<\/code>.<\/p>\n
\r\nif __name__ == \"__main__\":\r\n\r\n    #Add command line arguments to fetch import file and database name\r\n    parser = argparse.ArgumentParser()\r\n    #Command line argument to fetch import file. File name is taken as 'synthetic_data.zip' if no argument is specified\r\n    parser.add_argument(\"--import_file\", type=str, help=\"Provide the import file zip\",\r\n                        nargs='?', default=\"synthetic_data.zip\", const=0)\r\n    #Command line argument to fetch database name. Database name is taken as 'neo4j' if no argument is specified\r\n    parser.add_argument(\"--database_name\", type=str, help=\"Provide the name of the database\",\r\n                        nargs='?', default=\"neo4j\", const=0)\r\n    args = parser.parse_args()\r\n    IMPORT_FILE = args.import_file\r\n    database = args.database_name\r\n<\/pre>\n
The --database_name<\/code> argument is used to specify the name of the database that would be restored, and the --import_file<\/code> argument is used to locate the backup file which was created in the previous section of this post.<\/p>\n
4. Now, given that our backup folder is archived, it’s necessary to extract or unzip the archived folder before we can import it. To accomplish this, let’s first define a function to extract our archived file.<\/p>\n
\r\ndef unzip_file(IMPORT_FILE,extract_dir):\r\n    with ZipFile(IMPORT_FILE, 'r') as zip:\r\n        # printing all the contents of the zip file\r\n        zip.printdir()    \r\n        # extracting all the files\r\n        print('Extracting all the files now...')\r\n        zip.extractall(extract_dir)\r\n        print('Done!')\r\n<\/pre>\n
Here, the IMPORT_FILE<\/code> parameter is the zip file containing the neo4j backup, and extract_dir<\/code> is the name of the directory where  the archive will be extracted to.<\/p>\n
5. Next, Let’s call this function from within our main<\/code> function:<\/p>\n
\r\n#Get the directory into which the archive will be extracted\r\nextract_dir = os.path.splitext(IMPORT_FILE)[0]    \r\nunzip_file(IMPORT_FILE,extract_dir)\r\n<\/pre>\n
We now have our backup folder ready to be imported. Next, lets take a look at how we can import this backup data to a new database.<\/p>\n
 B. Importing the backup <\/h5>\n
1. Prior to restoring our database, it is necessary to clear any existing data to ensure accurate restoration. This can be achieved by executing a cypher query that clears the database. In order to run the query, we will be utilizing the Graph<\/code> library from py2neo<\/code>.<\/p>\n
2. First, let’s create a new file that will hold our cypher queries and name it “cypher.py”<\/em>. Then, add the following query to the file:<\/p>\n
\r\n#Delete all the records in the Database(WARNING: Never use this query in production database)\r\nDELETE_ALL_RECORDS = \"MATCH (n)\\\r\n                      DETACH DELETE n\"\r\n<\/pre>\n
3. Next, lets import this file in our python script<\/p>\n
\r\nimport cypher\r\n<\/pre>\n
4. Now that we have imported the Query, lets define a constant GRAPH<\/code> in our main<\/code> function ,which will hold the authentication for the Neo4j database that will enable us to run the query against the database<\/p>\n
\r\nGRAPH = auth()\r\n<\/pre>\n
5. Now lets define the auth()<\/code> function.<\/p>\n
\r\ndef auth():\r\n    \"Authenticating with the Database\"\r\n    GRAPH = None\r\n    try:\r\n        GRAPH = Graph(HOSTNAME, auth=(USERNAME, PASSWORD))\r\n        print(\"Database authenticated\")\r\n    except Exception as error:\r\n        raise RuntimeError('Database authentication failed') from error\r\n    return GRAPH\r\n<\/pre>\n
6. We can now run the query to clear the database in our main<\/code> function:<\/p>\n
\r\n#Clear the existing data in database\r\nclear_database = GRAPH.run(cypher.DELETE_ALL_RECORDS)\r\n<\/pre>\n
7. Having cleared our database, we can now proceed to restore the synthetic data. For this purpose, we need to define the connection settings for the Neo4j database and create a driver object using the GraphDatabase.driver()<\/code> function, which is the same approach used in the backup script.<\/p>\n
\r\nclear_database = GRAPH.run(cypher.DELETE_ALL_RECORDS)\r\nencrypted = False\r\ntrust = \"TRUST_ALL_CERTIFICATES\"\r\ndriver = GraphDatabase.driver(HOSTNAME, auth=(USERNAME, PASSWORD), encrypted=encrypted, trust=trust)\r\n<\/pre>\n
Note:<\/code> As mentioned previously, the encryption is set to false only because the database is on the local system. However, if the database exists on a remote server, make sure to set the encryption to true.<\/p>\n
8. Finally let’s import the synthetic data to the database by creating an instance of the Importer<\/code> class from the neo4j_backup<\/code> library and calling the import_data()<\/code> method.<\/p>\n
\r\n#Import the data to the database\r\nimporter = Importer(project_dir=extract_dir, driver=driver, database=database, input_yes= False)\r\nimporter.import_data()\r\n<\/pre>\n
C. Putting it all together<\/h5>\n
That\u2019s it! We are finally done with our coding. Our complete code should look similar to this:<\/p>\n
\r\n\"\"\"\r\nClear the existing database and restore the Neo4j backup \r\n\"\"\"\r\nimport os\r\nimport sys\r\nfrom neo4j import GraphDatabase\r\nfrom neo4j_backup import Importer\r\nfrom py2neo import Graph\r\nfrom decouple import config\r\nimport cypher\r\nfrom zipfile import ZipFile\r\nimport argparse\r\n\r\n# Grabbing environment variables\r\nHOSTNAME = config(\"DATABASE_HOST\")\r\nUSERNAME = config(\"DATABASE_USERNAME\")\r\nPASSWORD = config(\"DATABASE_PASSWORD\")\r\n\r\ndef auth():\r\n    \"Authenticating with the Database\"\r\n    GRAPH = None\r\n    try:\r\n        GRAPH = Graph(HOSTNAME, auth=(USERNAME, PASSWORD))\r\n        print(\"Database authenticated\")\r\n    except Exception as error:\r\n        raise RuntimeError('Database authentication failed') from error\r\n    return GRAPH\r\n\r\ndef unzip_file(IMPORT_FILE,extract_dir):\r\n    \"Unzip the import files\"\r\n    with ZipFile(IMPORT_FILE, 'r') as zip:\r\n        # printing all the contents of the zip file\r\n        zip.printdir()    \r\n        # extracting all the files\r\n        print('Extracting all the files now...')\r\n        zip.extractall(extract_dir)\r\n        print('Done!')\r\n\r\nif __name__ == \"__main__\":\r\n\r\n    #Add command line arguments to fetch import file and database name\r\n    parser = argparse.ArgumentParser()\r\n    #Command line argument to fetch import file. File name is taken as 'synthetic_data.zip' if no argument is specified\r\n    parser.add_argument(\"--import_file\", type=str, help=\"Provide the import file zip\",\r\n                        nargs='?', default=\"synthetic_data.zip\", const=0)\r\n    #Command line argument to fetch database name. Database name is taken as 'neo4j' if no argument is specified\r\n    parser.add_argument(\"--database_name\", type=str, help=\"Provide the name of the database\",\r\n                        nargs='?', default=\"neo4j\", const=0)\r\n    args = parser.parse_args()\r\n    IMPORT_FILE = args.import_file\r\n    database = args.database_name\r\n\r\n    #Get the directory into which the archive will be extracted\r\n    extract_dir = os.path.splitext(IMPORT_FILE)[0]    \r\n    unzip_file(IMPORT_FILE,extract_dir)\r\n\r\n    GRAPH = auth()\r\n\r\n    #Clear the existing data in database\r\n    clear_database = GRAPH.run(cypher.DELETE_ALL_RECORDS)\r\n    encrypted = False\r\n    trust = \"TRUST_ALL_CERTIFICATES\"\r\n    driver = GraphDatabase.driver(HOSTNAME, auth=(USERNAME, PASSWORD), encrypted=encrypted, trust=trust)\r\n\r\n    #Import the data to the database\r\n    importer = Importer(project_dir=extract_dir, driver=driver, database=database, input_yes=False)\r\n    importer.import_data()\r\n<\/pre>\n
Restoring and verifying the data<\/h4>\n
1. To run the restore script, first make sure that the Neo4j database you want to restore is up and running.
\n2. Open a terminal, navigate to the location of the neo4j_restore_script.py<\/em> file and run the following command:<\/p>\n
\r\npython neo4j_restore_script.py --database_name  --import_file \r\n<\/pre>\n
Make sure to replace database_name<\/code> with the name of the database you wish to restore the data into, and backup_file<\/code> with the the archived backup file that was created by the backup script.
\n
\"Execution<\/a>
Restoring synthetic data to Neo4j database<\/figcaption><\/figure><\/p>\n
3. Once the script finish running, check your database. If everything went smoothly, your  database should be populated with the synthetic data that we created.<\/p>\n
Hurray! We have successfully restored our synthetic data! <\/p>\n
Further reading<\/h3>\n
If you’re interested in checking out how we can setup tests for the synthetic data that we created you can checkout out the test “test_responses_between_dates.py<\/a><\/em>” on our Qxf2-survey<\/a> GitHub repository.
\nTo run the test, you will need to have the Qxf2-survey<\/a> app set up on your machine. You can follow the instructions provided in the Qxf2-survey app’s Github documentation to set up the survey app on your local system.<\/p>\n
\n
Hire technical testers from Qxf2<\/h3>\n
Qxf2 is the home of the technical tester. As you can see from this post, our testers go beyond the regular “manual” or “automation” paradigm of testing. We are engineers who test software. We lay the groundwork for testing, improve testability and enable your entire team to participate in testing. If you are looking for testers with solid engineering backgrounds, reach out to us here<\/a>.<\/p>\n
\n","protected":false},"excerpt":{"rendered":"
Qxf2 engineers are fans of using synthetic data for testing. We have used this technique for years now. But until recently, our experience was limited to SQL. We needed to work with neo4j in one of our projects. There was enough custom Python code and Cypher queries that we had to write as part of creating, backing up and restoring […]<\/p>\n","protected":false},"author":29,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[330,331,18,329],"tags":[],"class_list":["post-17608","post","type-post","status-publish","format-standard","hentry","category-neo4j","category-neo4j-backup","category-python","category-synthetic-data"],"_links":{"self":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/17608","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/users\/29"}],"replies":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/comments?post=17608"}],"version-history":[{"count":91,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/17608\/revisions"}],"predecessor-version":[{"id":17812,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/17608\/revisions\/17812"}],"wp:attachment":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/media?parent=17608"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/categories?post=17608"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/tags?post=17608"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}