AI Image & Video Generation

Think of it as:

• Text-to-Image: Giving a painter detailed instructions for a masterpiece

• Text-to-Video: Directing a movie scene with words alone

• AI Video Editing: Having an infinite team of visual effects artists

The AI Visual Generation Landscape

Major Players & Tools:

1. Stable Diffusion: The Open-Source Powerhouse

Stable Diffusion is the most popular open-source image generation model that you can run locally or via APIs.

Basic Stable Diffusion Usage:

python

# Using Diffusers library (Hugging Face)
from diffusers import StableDiffusionPipeline
import torch

# Load the model
pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32
)
pipe = pipe.to("cuda" if torch.cuda.is_available() else "cpu")

def generate_image(prompt, negative_prompt="", steps=20, guidance_scale=7.5):
    """Generate image from text prompt"""
    with torch.autocast("cuda" if torch.cuda.is_available() else "cpu"):
        image = pipe(
            prompt=prompt,
            negative_prompt=negative_prompt,
            num_inference_steps=steps,
            guidance_scale=guidance_scale,
            width=512,
            height=512
        ).images[0]
    
    return image

# Generate an image
image = generate_image(
    prompt="a majestic dragon flying over a medieval castle at sunset, fantasy art, highly detailed",
    negative_prompt="blurry, low quality, distorted",
    steps=25
)
image.save("dragon_castle.png")

Advanced Prompt Engineering for Images:

python

# Structured prompt templates
IMAGE_PROMPT_TEMPLATES = {
    'photography': """
    {subject}, {style} photography
    {lighting} lighting, {composition}
    {camera_settings}, {lens_type}
    professional, high resolution, 8K
    """,
    
    'art': """
    {subject}, {art_style} style
    by {artist_reference}
    {color_palette} color scheme
    {mood}, detailed, masterpiece
    """,
    
    'product': """
    product shot of {product}
    {background}, professional lighting
    clean, modern, commercial photography
    high detail, studio quality
    """
}

def build_photography_prompt(subject, style="cinematic", lighting="dramatic", 
                           composition="rule of thirds", camera="Sony A7III", lens="50mm f/1.8"):
    return f"""
    {subject}, {style} photography
    {lighting} lighting, {composition} composition
    shot on {camera} with {lens} lens
    professional, 8K, ultra detailed, sharp focus
    """

Key Parameters Explained:

python

# Optimal settings for different scenarios
GENERATION_CONFIGS = {
    'high_quality': {
        'steps': 30,
        'guidance_scale': 7.5,
        'width': 768,
        'height': 768,
        'sampler': 'DPMSolverMultistep'
    },
    'fast': {
        'steps': 15,
        'guidance_scale': 7.0,
        'width': 512,
        'height': 512,
        'sampler': 'EulerA'
    },
    'creative': {
        'steps': 50,
        'guidance_scale': 10.0,
        'width': 512,
        'height': 512,
        'sampler': 'DPM++ 2M Karras'
    }
}

2. ComfyUI: Advanced Visual Programming Interface

ComfyUI provides a node-based interface for complex Stable Diffusion workflows.

Why ComfyUI?

• Visual workflows with drag-and-drop nodes

• Infinite flexibility for complex pipelines

• Better performance and memory management

• Community workflows and sharing

Basic ComfyUI Workflow Structure:

python

# Pseudocode representing ComfyUI node workflow
workflow = {
    'load_checkpoint': {
        'type': 'LoadCheckpoint',
        'inputs': {'ckpt_name': 'model.safetensors'},
        'outputs': ['model', 'clip', 'vae']
    },
    
    'load_prompt': {
        'type': 'CLIPTextEncode',
        'inputs': {
            'text': 'a beautiful landscape',
            'clip': '@load_checkpoint.clip'
        },
        'outputs': ['conditioning']
    },
    
    'ksampler': {
        'type': 'KSampler',
        'inputs': {
            'model': '@load_checkpoint.model',
            'positive': '@load_prompt.conditioning',
            'negative': '@load_negative_prompt.conditioning',
            'latent_image': '@empty_latent.latent',
            'steps': 20,
            'cfg': 7.5
        },
        'outputs': ['latent']
    },
    
    'vae_decode': {
        'type': 'VAEDecode',
        'inputs': {
            'samples': '@ksampler.latent',
            'vae': '@load_checkpoint.vae'
        },
        'outputs': ['image']
    },
    
    'save_image': {
        'type': 'SaveImage',
        'inputs': {'images': '@vae_decode.image'},
        'outputs': []
    }
}

Advanced ComfyUI Workflows:

python

# Workflow for image-to-image with control nets
advanced_workflow = {
    'load_image': {
        'type': 'LoadImage',
        'inputs': {'image_path': 'input.jpg'},
        'outputs': ['image']
    },
    
    'preprocess_pose': {
        'type': 'OpenPosePreprocessor',
        'inputs': {'image': '@load_image.image'},
        'outputs': ['pose_map']
    },
    
    'apply_controlnet': {
        'type': 'ControlNetApply',
        'inputs': {
            'conditioning': '@load_prompt.conditioning',
            'control_net': '@load_controlnet.controlnet',
            'image': '@preprocess_pose.pose_map'
        },
        'outputs': ['controlled_conditioning']
    },
    
    'img2img_sampler': {
        'type': 'KSampler',
        'inputs': {
            'model': '@load_checkpoint.model',
            'positive': '@apply_controlnet.controlled_conditioning',
            'latent_image': '@vae_encode.latent',
            'denoise': 0.75
        },
        'outputs': ['latent']
    }
}

ComfyUI API Integration:

python

import requests
import json
import io
from PIL import Image

class ComfyUIAPI:
    def __init__(self, server_url="http://127.0.0.1:8188"):
        self.server_url = server_url
    
    def generate_image(self, workflow, prompt, output_node="save_image"):
        """Execute a ComfyUI workflow"""
        # Prepare the prompt
        prompt_data = self._build_prompt(workflow, prompt)
        
        # Queue the prompt
        response = requests.post(
            f"{self.server_url}/prompt",
            json={"prompt": prompt_data}
        )
        prompt_id = response.json()['prompt_id']
        
        # Wait for completion and get image
        return self._get_result(prompt_id, output_node)
    
    def _build_prompt(self, workflow, positive_prompt, negative_prompt=""):
        """Build ComfyUI prompt JSON"""
        prompt_data = {}
        
        for node_id, node_config in workflow.items():
            prompt_data[node_id] = {
                "class_type": node_config['type'],
                "inputs": self._resolve_inputs(node_config['inputs'])
            }
        
        return prompt_data
    
    def queue_workflow(self, workflow_config):
        """Queue a predefined workflow"""
        with open('workflow_api.json', 'r') as f:
            workflow = json.load(f)
        
        return self.generate_image(workflow, workflow_config['prompt'])

# Usage
comfy = ComfyUIAPI()
result = comfy.generate_image(my_workflow, "a cyberpunk city at night")

3. Runway ML: Professional Video Generation

Runway provides state-of-the-art video generation and editing tools.

Basic Runway API Usage:

python

import requests
import time

class RunwayML:
    def __init__(self, api_key):
        self.api_key = api_key
        self.base_url = "https://api.runwayml.com/v1"
    
    def generate_video(self, prompt, duration=4, resolution="1024x576"):
        """Generate video from text prompt"""
        response = requests.post(
            f"{self.base_url}/video/generations",
            headers={"Authorization": f"Bearer {self.api_key}"},
            json={
                "prompt": prompt,
                "duration": duration,
                "resolution": resolution,
                "model": "gen-2"
            }
        )
        
        generation_id = response.json()['id']
        return self._wait_for_completion(generation_id)
    
    def image_to_video(self, image_path, prompt, motion_strength=0.5):
        """Convert image to video with motion"""
        # Upload image
        with open(image_path, 'rb') as f:
            upload_response = requests.post(
                f"{self.base_url}/files",
                headers={"Authorization": f"Bearer {self.api_key}"},
                files={"file": f}
            )
        image_id = upload_response.json()['id']
        
        # Generate video from image
        response = requests.post(
            f"{self.base_url}/video/generations",
            headers={"Authorization": f"Bearer {self.api_key}"},
            json={
                "prompt": prompt,
                "image": image_id,
                "motion_strength": motion_strength,
                "model": "gen-2"
            }
        )
        
        return self._wait_for_completion(response.json()['id'])
    
    def _wait_for_completion(self, generation_id, timeout=300):
        """Wait for generation to complete"""
        start_time = time.time()
        while time.time() - start_time < timeout:
            response = requests.get(
                f"{self.base_url}/video/generations/{generation_id}",
                headers={"Authorization": f"Bearer {self.api_key}"}
            )
            
            status = response.json()['status']
            if status == 'succeeded':
                return response.json()['output']['url']
            elif status == 'failed':
                raise Exception("Generation failed")
            
            time.sleep(5)
        
        raise TimeoutError("Generation timeout")

# Usage
runway = RunwayML(api_key="your_api_key")
video_url = runway.generate_video(
    prompt="A astronaut floating in space, galaxy in background, cinematic, 4K",
    duration=5,
    resolution="1280x720"
)

Video Prompt Engineering:

python

VIDEO_PROMPT_TEMPLATES = {
    'cinematic': """
    {subject}, {scene_description}
    {camera_movement}, {shot_type}
    cinematic, film grain, {color_grade}
    {style_reference}, 4K, 24fps
    """,
    
    'animation': """
    {character} {action}, {environment}
    {animation_style} animation
    smooth motion, detailed characters
    vibrant colors, {art_director} style
    """,
    
    'motion_graphics': """
    {concept} explanation, motion graphics
    clean, modern, corporate style
    smooth animations, {color_scheme}
    professional, 2D/3D hybrid
    """
}

def build_video_prompt(subject, camera_movement="slow dolly", shot_type="medium shot",
                      style="cinematic", color_grade="teal and orange"):
    return f"""
    {subject}, {style} style
    {camera_movement} camera movement, {shot_type}
    {color_grade} color grading
    high detail, 4K resolution, 24fps
    professional cinematography
    """

4. Pika Labs: Text-to-Video Specialization

Pika focuses specifically on high-quality text-to-video generation.

Pika-like Video Generation:

python

class PikaVideoGenerator:
    def __init__(self, config):
        self.config = config
    
    def create_scene_sequence(self, script):
        """Convert script to video scenes"""
        scenes = self._parse_script(script)
        generated_clips = []
        
        for i, scene in enumerate(scenes):
            print(f"Generating scene {i+1}/{len(scenes)}: {scene['description']}")
            
            clip = self._generate_scene(
                scene['description'],
                scene.get('style', 'cinematic'),
                scene.get('duration', 3)
            )
            generated_clips.append(clip)
        
        return self._compile_video(generated_clips)
    
    def _parse_script(self, script):
        """Parse narrative script into individual scenes"""
        # Simple scene detection (in practice, use NLP)
        scenes = []
        current_scene = {"description": "", "duration": 3}
        
        for line in script.split('\n'):
            line = line.strip()
            if line.startswith('SCENE') or len(current_scene['description']) > 200:
                if current_scene['description']:
                    scenes.append(current_scene)
                    current_scene = {"description": "", "duration": 3}
                current_scene['description'] = line
            elif line:
                current_scene['description'] += " " + line
        
        if current_scene['description']:
            scenes.append(current_scene)
        
        return scenes

# Usage
script = """
SCENE 1: A spaceship landing on an alien planet at dusk
The ship descends slowly through purple clouds, lights blinking

SCENE 2: Astronaut stepping out onto the surface
The astronaut takes first steps, looking around in wonder

SCENE 3: Alien landscape reveal
Strange plants glowing with bioluminescence, two moons in sky
"""

pika = PikaVideoGenerator(config={})
video = pika.create_scene_sequence(script)

5. Advanced Image & Video Workflows

Multi-Stage Image Generation:

python

class AdvancedImageWorkflow:
    def __init__(self):
        self.stages = []
    
    def add_controlnet_stage(self, controlnet_type, image_reference, strength=1.0):
        """Add controlnet guidance stage"""
        self.stages.append({
            'type': 'controlnet',
            'controlnet': controlnet_type,
            'reference': image_reference,
            'strength': strength
        })
    
    def add_upscale_stage(self, scale_factor=2, method="ESRGAN"):
        """Add image upscaling stage"""
        self.stages.append({
            'type': 'upscale',
            'scale': scale_factor,
            'method': method
        })
    
    def add_inpainting_stage(self, mask, prompt):
        """Add inpainting stage"""
        self.stages.append({
            'type': 'inpaint',
            'mask': mask,
            'prompt': prompt
        })
    
    def execute_workflow(self, initial_prompt, initial_image=None):
        """Execute multi-stage workflow"""
        current_image = initial_image
        
        for i, stage in enumerate(self.stages):
            print(f"Executing stage {i+1}: {stage['type']}")
            
            if stage['type'] == 'controlnet':
                current_image = self._apply_controlnet(
                    current_image, stage['controlnet'], 
                    stage['reference'], stage['strength']
                )
            elif stage['type'] == 'upscale':
                current_image = self._upscale_image(
                    current_image, stage['scale'], stage['method']
                )
            elif stage['type'] == 'inpaint':
                current_image = self._inpaint_image(
                    current_image, stage['mask'], stage['prompt']
                )
        
        return current_image

# Usage
workflow = AdvancedImageWorkflow()
workflow.add_controlnet_stage("openpose", reference_pose_image)
workflow.add_upscale_stage(scale_factor=2)
workflow.add_inpainting_stage(mask_image, "beautiful background")

result = workflow.execute_workflow(
    "a person dancing in a studio", 
    initial_image=base_image
)

Video Editing Pipeline:

python

class VideoEditingPipeline:
    def __init__(self):
        self.clips = []
        self.transitions = []
        self.effects = []
    
    def add_text_to_video(self, video_clip, text, position, duration):
        """Add animated text to video"""
        # Implementation for adding text overlays
        pass
    
    def apply_color_grading(self, video_clip, lut_path):
        """Apply color grading LUT"""
        # Implementation for color grading
        pass
    
    def generate_slow_motion(self, video_clip, slow_factor=2):
        """Generate slow motion effect"""
        # Implementation for frame interpolation
        pass
    
    def composite_video(self, background_clip, foreground_clip, mask):
        """Composite multiple video layers"""
        # Implementation for video compositing
        pass

    def export_final_video(self, output_path, format="mp4", quality="high"):
        """Export final edited video"""
        # Implementation for video export
        pass

6. Practical Applications & Use Cases

Content Creation Pipeline:

python

class ContentCreationStudio:
    def __init__(self):
        self.image_generator = StableDiffusionPipeline.from_pretrained(...)
        self.video_editor = VideoEditingPipeline()
    
    def create_social_media_post(self, topic, platform="instagram"):
        """Create complete social media post"""
        # Generate relevant image
        image_prompt = self._build_social_media_prompt(topic, platform)
        image = self.image_generator.generate_image(image_prompt)
        
        # Generate short video clip
        video_prompt = self._build_video_prompt(topic)
        video_clip = self.video_generator.generate_video(video_prompt)
        
        # Add branding and text
        branded_video = self._add_branding(video_clip, platform)
        
        return {
            'image': image,
            'video': branded_video,
            'caption': self._generate_caption(topic)
        }
    
    def create_product_demo(self, product_description, duration=30):
        """Create product demonstration video"""
        scenes = [
            "Product reveal with dramatic lighting",
            "Close-up shots showing features",
            "Product in use scenario",
            "Benefits and call to action"
        ]
        
        generated_clips = []
        for scene in scenes:
            clip = self.video_generator.generate_video(
                f"{product_description}, {scene}, professional product photography"
            )
            generated_clips.append(clip)
        
        return self.video_editor.compile_video(generated_clips)

Real-Time Image Generation API:

python

from fastapi import FastAPI, UploadFile, File
from fastapi.responses import StreamingResponse
import io

app = FastAPI()

@app.post("/generate-image")
async def generate_image_endpoint(prompt: str, style: str = "realistic"):
    """API endpoint for image generation"""
    image = generate_image(prompt, style)
    
    # Convert to bytes for response
    img_byte_arr = io.BytesIO()
    image.save(img_byte_arr, format='PNG')
    img_byte_arr.seek(0)
    
    return StreamingResponse(img_byte_arr, media_type="image/png")

@app.post("/image-to-video")
async def image_to_video_endpoint(
    image: UploadFile = File(...),
    prompt: str = "",
    motion: float = 0.5
):
    """Convert uploaded image to video"""
    image_data = await image.read()
    # Process image and generate video
    video_url = runway.image_to_video(image_data, prompt, motion)
    
    return {"video_url": video_url, "status": "completed"}

Key Takeaways

1. Stable Diffusion is the foundation for open-source image generation

2. ComfyUI provides unparalleled control through visual programming

3. Runway ML offers professional-grade video generation tools

4. Prompt engineering is crucial for consistent, high-quality results

5. Workflow automation enables complex multi-stage generations

6. Real-world applications span from content creation to product demos

Start with simple text-to-image generations, then progressively add complexity with control nets, inpainting, and eventually video generation as you become comfortable with the tools and workflows!

The AI visual generation space is moving incredibly fast, with new models and capabilities emerging weekly. The key is to build flexible workflows that can adapt to new tools while maintaining consistent quality output.