Hands-On Guide to Generating Synthetic Data with Gretel AI

Key Features of Gretel’s Synthetic Data Tools

Here are some features that make Gretel a preferred choice for developers:

• Privacy-First Approach: Generate data without exposing sensitive information.
• Customizable Models: Fine-tune parameters to align with specific use cases.
• Cloud Integration: Train models effortlessly using Gretel’s cloud platform.
• Evaluation Reports: Measure the statistical alignment between real and synthetic datasets.

Hands-On Implementation

Step 1: Setting Up the Environment

Start by installing the required dependencies and configuring the Gretel API session:

!pip install -Uqq gretel-client==0.24.1
from gretel_client import configure_session
pd.set_option("max_colwidth", None)
configure_session(api_key="prompt", cache="yes", validate=True)

Step 2: Loading the Dataset

Download and preview your dataset:

import pandas as pd

DATASET_PATH = "https://huggingface.co/api/datasets/bitext/Bitext-customer-support-llm-chatbot-training-dataset/parquet/default/train/0.parquet"

df = pd.read_parquet(DATASET_PATH)
print(df.head())

Step 3: Initializing the Project

Create or retrieve a unique project to manage your synthetic data pipeline:

from gretel_client.projects import create_or_get_unique_project
project = create_or_get_unique_project(name="synthetic-data")

Step 4: Configuring the Synthetic Model

Customize the ACTGAN model for tabular data synthesis:

from gretel_client.projects.models import read_model_config
import json

config = read_model_config("synthetics/tabular-actgan")
config["models"][0]["actgan"]["params"]["epochs"] = "auto"
config["models"][0]["actgan"]["generate"]["num_records"] = 10000
print(f"Model configuration:\n{json.dumps(config, indent=2)}")

print(df.head())

Output

Step 5: Training the Model

Train the ACTGAN model using Gretel’s cloud infrastructure:

from gretel_client.helpers import poll

model = project.create_model_obj(model_config=config, data_source=DATASET_PATH)

model.submit_cloud()

poll(model, verbose=False)

Step 6: Retrieving Synthetic Data

Access the generated synthetic dataset:

import requests

artifact_link = model.get_artifact_link("data_preview")
response = requests.get(artifact_link)

# Decompress and preview the dataset
import gzip, io
with gzip.GzipFile(fileobj=io.BytesIO(response.content)) as f:

    synthetic_df = pd.read_parquet(io.BytesIO(f.read()))

print(synthetic_df.head())

Step 7: Generating Data Quality report

Let’s Generate report that shows the statistical performance between the training and synthetic data

import IPython

from smart_open import open

IPython.display.HTML(data=open(model.get_artifact_link("report")).read(), metadata=dict(isolated=True))

The correlation difference between the training data and the synthetic data is minimal, which can be clearly seen in the image below.

Challenges and Best Practices

Common Challenges

• Dataset Quality: The effectiveness of synthetic data relies heavily on the quality of the input dataset.
• Hyperparameter Tuning: Adjusting model parameters for optimal results can be time-consuming.
• Data Validation: Ensuring the synthetic data matches the real-world data’s statistical properties requires rigorous evaluation.

Best Practices

• Preprocess Data: Clean and normalize input data for consistent results.
• Use Evaluation Tools: Leverage Gretel’s built-in reports to validate data quality.
• Experiment Iteratively: Test different configurations to fine-tune the output.

Final Thoughts

Synthetic data generation is a game-changer for data-driven workflows, enabling innovation while addressing privacy concerns. Gretel AI simplifies this process with its user-friendly tools and robust capabilities. Whether you’re augmenting datasets for machine learning or anonymizing sensitive data, Gretel offers a scalable solution for diverse use cases.

References

• Gretel AI Official Documentation
• Gretel's Github Repo