Personal R&D · Generative AI · 3D

Teaching a visual style, then giving sketches physical form.

Two connected experiments: training custom SDXL LoRAs on Modal, and transforming hand-drawn characters into textured GLB assets through OpenAI image generation and Hunyuan 3D.

RoleDataset design, prompting, training workflow, visual evaluation

Model stackSDXL, Diffusers, LoRA, OpenAI image generation, Hunyuan 3D

ComputeModal with NVIDIA L40S and A10G GPUs

OutputsLoRA weights, generated samples and textured GLB characters

Orange dog character rendered from a hand-drawn sketch

Sketch → render → 3D

Part 01Training my own LoRAOriginal drawings → Modal GPU → a reusable visual style Part 02Turning a sketch into 3DSketch → OpenAI render → Hunyuan textured GLB

PART 01

LoRA on Modal

Experiment one · style learning

Could a small dataset teach SDXL a recognisable crayon language?

I drew and labelled 34 images, then trained a custom SDXL LoRA on Modal. The goal was style learning rather than memorising one character: waxy texture, uneven contours, bright colour blocks and the spatial logic of a child’s drawing across new subjects.

First contact sheet of Niharika's original crayon drawings — Original drawings · sheet 1 of 3

Second contact sheet of Niharika's original crayon drawings — Original drawings · sheet 2 of 3

Third contact sheet of Niharika's original crayon drawings — Original drawings · sheet 3 of 3

Dataset

The complete corpus: 34 drawings made by me

These three contact sheets show every original drawing used for the experiment. Each image received a descriptive caption anchored by one shared trigger token.

crayonstress_style

22: single-dog images
11: multi-subject or human scenes
1: non-dog style reference

“crayonstress_style, a white puppy sitting at a table with a slice of cake and a cup”

Base modelSDXL 1.0

GPUL40S

Resolution768 × 768

Rank8

Steps600

Learning rate5 × 10⁻⁵

Generated crayon drawing of a dog walking toward colourful mountains — Mountain dog

Generated crayon drawing of two dogs near a red house — Two dogs and a house

Generated crayon drawing of a butterfly over a flowered hillside — Butterfly hillside

Generated crayon drawing of a yellow animatronic dog — Animatronic dog

PART 01B

LoRA scale test

A second LoRA tested a larger, noisier visual archive.

This dataset started with 4,977 archival motif crops. The preparation workflow balanced source pages, rejected tiny and extreme-aspect images, removed perceptual duplicates and padded each selected motif to a 768-pixel square.

Contact sheet of archival Indian decorative motifs — 600 prepared examples sampled across source-page groups.

Generated archival-style dog emblem — Sample, LoRA scale 1.0

Generated archival-style cat emblem — Sample, LoRA scale 1.0

Early read

The muted paper tone, dense ink contours and framed emblem composition transferred consistently. Broader prompt and LoRA-scale testing would be needed before judging semantic accuracy.

PART 02

Sketch to 3D asset

What happens when a hand-drawn character becomes a model you can hold, rotate and reuse?

PART 02

Asset workflow

Experiment two · form generation

The 2D render acts as a bridge between drawing and geometry.

A sketch contains the identity, but not enough spatial information for predictable 3D generation. I first used OpenAI image generation to clarify volume and material, then passed that reference to Hunyuan to create the mesh and texture.

Workflow diagram

Sketch → model → textured asset

01Original sketch
Silhouette, palette and character cues.
02OpenAI render
Volume, lighting and a clean reference view.
03Hunyuan geometry
A single-mesh 3D model generated from the image.
04Textured GLB
One material with an embedded 2K colour atlas.

Failed experiment · useful finding

Why not send the flat drawing directly to 3D?

My first dog model was generated straight from the flat 2D drawing. The result tried to interpret graphic marks as physical form without enough evidence about depth, volume or the hidden side of the character.

That “bad dog” made the missing step clear: a 3D-style reference render gives Hunyuan stronger information about rounded forms, material, lighting and spatial relationships before it builds the mesh.

Flat sketchAmbiguous depth3D-style renderClearer volume

Flat hand-drawn orange dog used in the direct-to-3D test — Input · flat 2D sketch

Output · direct conversion (“bad dog”)

Interactive result

Inspect the Hunyuan GLB outputs.

Drag to rotate and scroll to zoom. Compare the textured export with the geometry-only model, then switch between the dog and butterfly.

Loading 3D model…

TEXTURED GLBDRAG · ZOOM · ROTATE

OUTPUTS

Two characters

Dog

Recognisable cues survive the regularisation.

The loose sketch becomes a clearer toy-like form while keeping the hat, blue bandana, orange body and walking pose.

Original hand-drawn orange dog — Original sketch

Polished orange dog reference render — OpenAI reference render

Butterfly · updated texture

The ChatGPT-guided texture carries the identity.

The geometry is simple. The updated textured GLB uses the ChatGPT-rendered reference to carry the orange, blue and turquoise marks across the wings and body.

Original hand-drawn colourful butterfly — Original sketch

Polished colourful butterfly reference render — OpenAI reference render

Future goal

Build my own automated sketch-to-asset pipeline.

Next, I hope to train my own LoRA with a larger and more varied collection of sketches, then automate the journey from a generated character to a 3D-style reference and a textured Hunyuan asset. The goal is a repeatable system for creating many original, production-ready 3D characters—not just one model at a time.

Return to the model viewer