How to Get Started with Apache Flink in 2026

Introduction

Apache Flink is an open-source distributed data processing framework that excels at both streaming and batch workloads in a unified way. Unlike tools like Spark that separate these paradigms, Flink treats data as an infinite continuous stream, with robust state management, low latency, and exactly-once semantics. In 2026, Flink powers real-time data pipelines at Netflix, Uber, and Alibaba, handling terabytes per second.

This beginner tutorial guides you through installing Flink locally via PyFlink (Python API), running a batch WordCount job, and a streaming job with socket input. You'll learn the basics: execution environment, DataStream API, transformations, and submission. By the end, you'll be ready to scale to a cluster. Every example is complete and runnable—just copy-paste onto your machine. Think of Flink like an industrial conveyor belt: data flows in, gets transformed in parallel, and comes out processed without loss.

Prerequisites

Java 11+ installed (OpenJDK recommended: check with java -version)
Python 3.9+ (with pip: python --version)
Unix-like system (Linux/Mac) or WSL on Windows
Terminal access and code editor (VS Code)
Basic Python knowledge (lists, functions)

Install PyFlink

install.sh

#!/bin/bash

# Check Java
java -version

# Update pip
python -m pip install --upgrade pip

# Install PyFlink (stable 2026 version)
pip install apache-flink==1.18.0

# Verify installation
python -c "import pyflink; print('PyFlink installed successfully')"

This script checks Java (required for Flink's JVM runtime), updates pip, and installs PyFlink. It's standalone and runs in one command. Stick to stable versions for reliability; test the import to confirm no errors.

Verify Your Environment

Run the install.sh script in a terminal. You should see the Java version (11+), PyFlink installation without warnings, and the confirmation message. PyFlink spins up a local JVM mini-cluster for each job, hiding deployment complexity. No need to download Flink binaries to get started.

First Batch Job: WordCount

batch_wordcount.py

from pyflink.datastream import StreamExecutionEnvironment
from pyflink.common.typeinfo import Types
from pyflink.datastream.functions import RuntimeContext

# Create the execution environment (batch by default with finite collection)
env = StreamExecutionEnvironment.get_execution_environment()

# Input data: list of sentences
text = ['Hello Flink', 'Welcome to Flink', 'Flink is fast', 'Apache Flink']

# Source: from_collection (batch-like)
ds = env.from_collection(
    collection=text,
    type_info=Types.STRING()
)

# Transformations: flat_map (split words) -> key_by (group by word) -> sum
# FlatMap to split
def split(line):
    if line:
        for word in line.lower().split():
            yield word, 1

ds = ds.flat_map(split, Types.TUPLE([Types.STRING(), Types.INT()]))

# Group by word and sum counts
ds.key_by(lambda x: x[0])\
  .sum(1)\
  .print()

# Execute the job
env.execute('Batch WordCount Job')

This batch job processes a finite collection of sentences: splits into words, counts by key with key_by and sum. The local environment uses 1 task manager by default. Copy-paste and run it; output shows words and counts (e.g., ('flink', 3)). Pitfall: Forgetting type_info causes serialization errors.

Run the Batch Job

run_batch.sh

#!/bin/bash

# Create Python file if not already done
cat > batch_wordcount.py << EOF
[INSERT THE PYTHON CODE ABOVE HERE - OMITTED FOR BREVITY IN PROMPT]
EOF

# Run the job locally
python batch_wordcount.py

# Expected output:
# ('apache', 1)
# ('fast', 1)
# ('flink', 3)
# etc.

This script creates and runs the job. It starts an ephemeral local cluster (JVM + Python). Watch the logs for parallelism=1. To scale, add env.set_parallelism(4). Avoid running without Java in PATH.

Understanding the Batch Job

The job simulates batch processing on finite data. flat_map emits (word, 1) tuples, key_by groups them, and sum aggregates. Output goes to the console. Analogy: like a MapReduce reduce, but stream-based. Add env.set_parallelism(2) to see distribution.

Streaming Job: Socket WordCount

stream_wordcount.py

from pyflink.datastream import StreamExecutionEnvironment, TimeCharacteristic
from pyflink.common.time import Time

env = StreamExecutionEnvironment.get_execution_environment()
env.set_stream_time_characteristic(TimeCharacteristic.EventTime)

# Streaming source: socket localhost:9999
stream = env.socket_text_stream('localhost', 9999, delimiter='\n')

# Same transformations
stream.flat_map(lambda line: [(word, 1) for word in line.lower().split() if line])\
      .key_by(lambda x: x[0])\
      .sum(1)\
      .print()

env.execute('Streaming WordCount')

This job reads an infinite stream from a TCP socket. Use EventTime for timestamps. Transformations are identical to batch, proving Flink's unification. Start the socket producer first; counts update live. Pitfall: Socket startup delay causes timeouts.

Start the Socket Producer

start_nc.sh

#!/bin/bash

# Install netcat if needed (brew install netcat or apt install netcat)

# Listen on port 9999, send lines
nc -lk 9999

# In another terminal, type sentences like:
# Hello Flink
# Flink rocks
# Ctrl+C to stop

Netcat simulates a streaming producer. Run it first, then the Python job. Type lines and watch live counts (e.g., ('flink', 2)). Stop with Ctrl+C. On Windows, use ncat or PowerShell.

Run the Streaming Job

run_stream.sh

#!/bin/bash

# Ensure nc is running (separate terminal)

# Launch the streaming job
python stream_wordcount.py

# Type in nc: words, watch output:
# ('hello', 1)
# ('flink', 5)
# Job runs until Ctrl+C

Combines producer + consumer. Shows low latency (<1s). For production, swap socket for Kafka. Avoid closing nc first; otherwise, EOF error.

Best Practices

Always specify types (Types.STRING(), etc.) to avoid serialization RuntimeErrors.
Set parallelism explicitly: env.set_parallelism(4) to simulate a cluster.
Handle exceptions in functions with try/except for resilience.
Use checkpoints for fault-tolerance: env.enable_checkpointing(5000).
Test locally before clusters; log with logging.getLogger().

Common Errors to Avoid

Java not found: Set JAVA_HOME and PATH; check java -version.
Socket not connected: Start nc BEFORE the job; use localhost exactly.
Missing types: Flink requires TypeInformation; causes TypeInformationExtractionException.
No yield in flat_map: Use generator (yield) for multi-emissions.

Next Steps

Official docs: Flink Docs
Cluster tutorial: Download Flink binaries for standalone mode.
Advanced: Kafka connector, SQL API.
Learni Training: Master Flink in production with our certified courses.