Scikit-Learn Tutorial: Build and Evaluate Your First Model

Getting a model to produce a prediction can feel like the finish line. Then a more useful question arrives: how would I know whether that prediction deserves any confidence? A few lines of training code cannot answer that on their own.

This scikit-learn tutorial builds a small classification model while keeping the evaluation honest. You will split the data before preprocessing, use a pipeline, compare a simple baseline and inspect mistakes. Those habits matter long after you outgrow the example.

The complete example uses scikit-learn’s bundled wine dataset, so it does not need a private customer dataset or a separate data download. It runs on a CPU. The code was checked with scikit-learn 1.8.0; results can vary across package versions and environments.

What you will build

The dataset contains 178 samples, 13 numeric features and three target classes. The task is to predict a class from the measurements. It is a small, well-known teaching dataset, useful for understanding the workflow rather than proving that a business application is ready.

Here, X contains the input measurements and y contains the class labels. A classifier learns from labeled examples. It will not invent a meaningful target for an unlabeled spreadsheet, and a dataset’s column names alone do not establish that a prediction is useful.

1. Create a separate Python environment

Use a supported Python 3 version compatible with the current scikit-learn release. A virtual environment keeps the tutorial’s packages separate from your system and other projects. On Linux or macOS:

mkdir sklearn-first-model
cd sklearn-first-model
python3 -m venv .venv
. .venv/bin/activate
python -m pip install scikit-learn

On Windows PowerShell, you can use the environment’s Python directly without changing script-execution policy:

mkdir sklearn-first-model
cd sklearn-first-model
py -m venv .venv
.\.venv\Scripts\python.exe -m pip install scikit-learn

The installation step requires network access to the package index or your organization’s approved mirror. Check the official installation guidance if your Python version is unsupported. The example itself uses local bundled data after installation.

2. Keep training information away from the test set

The code holds back 20 percent of the data for a final test. A fixed random_state makes the split repeatable for this example. stratify=y keeps class proportions approximately similar across the two subsets.

The important boundary comes before scaling. If you calculate means and standard deviations using the full dataset, the training procedure has already learned something about the test inputs. That is data leakage, even if you never pass the test labels to the classifier.

A scikit-learn pipeline fits the scaler and classifier together. During cross-validation, each fold learns its own preprocessing from that fold’s training portion. The held-out test set stays outside that development loop.

3. Run the complete scikit-learn pipeline

Save the following code in a file named first_model.py inside the project directory. Logistic regression is used as a straightforward classification model; despite its name, it is appropriate for this classification example.

import sklearn
from sklearn.datasets import load_wine
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    classification_report,
    confusion_matrix,
    f1_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_score,
    train_test_split,
)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

data = load_wine()
X, y = data.data, data.target

# Split before learning any preprocessing parameters.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

model = make_pipeline(
    StandardScaler(),
    LogisticRegression(max_iter=2000),
)

# Inspect development performance using training data only.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_scores = cross_val_score(
    model, X_train, y_train, cv=cv, scoring="f1_macro"
)

# Fit the chosen workflow, then evaluate the held-out test set.
model.fit(X_train, y_train)
predictions = model.predict(X_test)

baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(X_train, y_train)
baseline_predictions = baseline.predict(X_test)

print("scikit-learn:", sklearn.__version__)
print("Train/test rows:", len(y_train), len(y_test))
print(f"CV macro F1: {cv_scores.mean():.3f} +/- {cv_scores.std():.3f}")
print(f"Baseline accuracy: {accuracy_score(y_test, baseline_predictions):.3f}")
print(f"Test accuracy: {accuracy_score(y_test, predictions):.3f}")
print(f"Test macro F1: {f1_score(y_test, predictions, average='macro'):.3f}")
print(classification_report(
    y_test, predictions, target_names=data.target_names,
    digits=3, zero_division=0
))
print("Confusion matrix (rows=true, columns=predicted):")
print(confusion_matrix(y_test, predictions))

With the Linux or macOS virtual environment activated, run:

python first_model.py

On Windows PowerShell, run the same file through the environment’s Python:

.\.venv\Scripts\python.exe first_model.py

4. Read the result without overclaiming

One checked run with scikit-learn 1.8.0 produced 142 training rows and 36 test rows. The summary was:

CV macro F1: 0.980 +/- 0.027
Baseline accuracy: 0.389
Test accuracy: 0.972
Test macro F1: 0.971

The cross-validation variation shown here is the standard deviation across five folds, not a confidence interval. The baseline always predicts the most frequent training class. It is deliberately simple: a useful starting comparison before judging the more elaborate workflow.

On that run, the confusion matrix was:

[[12  0  0]
 [ 0 14  0]
 [ 0  1  9]]

Rows represent true classes and columns represent predicted classes, ordered as class 0, class 1 and class 2. The final row shows one class-2 example predicted as class 1. That single error matters when interpreting the per-class report.

Accuracy is the share of correct predictions. Precision asks how often predictions of a class were correct; recall asks how many actual examples of that class were found. F1 combines precision and recall. Macro F1 gives each class equal weight, making it useful alongside accuracy when class-level behavior matters.

A test set of 36 rows is small. One additional mistake would noticeably change the result. None of these numbers establishes performance on future business data, different measurement processes or populations absent from the teaching dataset.

5. Understand why scaling lives inside the pipeline

The numeric features have different scales. StandardScaler estimates a mean and scale from the training inputs, then applies the learned transformation to later inputs. It does not need to be manually refitted before each prediction.

At prediction time, pass inputs in the same feature order and units used during training. Calling fit on new inputs changes the learned workflow; calling predict uses it. Keep the preprocessing with the model so the two cannot quietly fall out of agreement.

6. Improve the experiment without turning the test into training

The code deliberately uses one chosen model setup. If you explore alternatives, compare them using training-set cross-validation. Keep preprocessing inside the pipeline and reserve the held-out set for the final assessment.

• Decide which error matters before choosing a headline metric.
• Compare a small number of justified candidates rather than chasing a score blindly.
• Inspect mistakes and check whether the input or label is questionable.
• Record package versions, the split strategy and the final evaluation decision.

If you inspect the final test result and repeatedly adjust the workflow to improve it, that set is no longer an untouched test. You need a new evaluation plan. More parameter searches do not automatically produce more trustworthy evidence.

When a random train-test split is the wrong choice

Real datasets often contain relationships that this demonstration does not model. Several records may belong to the same customer, device or document. Splitting related examples across training and testing can exaggerate performance.

For grouped data, use a split that respects group boundaries. For forecasting or time-dependent deployment, evaluate in a way that respects time and prevents future information from entering training. The right test imitates the conditions under which the model will actually be used.

Common problems and practical fixes

• ModuleNotFoundError: install scikit-learn with the same Python executable that runs the script.
• Python compatibility error: check the release’s installation requirements; do not force an unsupported combination.
• Different numbers: check versions and that the code, split and dataset are unchanged.
• Convergence warning: verify scaling and input quality, then investigate solver settings rather than hiding the warning.
• Suspiciously perfect results: check duplicates, target-derived features and preprocessing leakage before celebrating.

Frequently asked questions

Do I need a GPU for this tutorial?

No. The dataset and model are small enough for an ordinary CPU. Begin with a working evaluation process before investing in specialized hardware.

Can I replace the bundled data with a CSV?

Yes, but you must define a legitimate target, select usable features and handle missing or categorical values appropriately. Do not simply pass every column to this example, especially identifiers or fields created after the outcome.

Is this model ready for production?

No. Production use needs suitable data, validation, access controls, deployment testing, monitoring and clear ownership. This tutorial teaches the learning and evaluation loop, which is one part of that work.

Build the habit of asking what the score means

Your first model does not need to be impressive. It needs a workflow you can explain: what it learned from, what stayed unseen, what it was compared with and where it made mistakes.

If you tried this process on a dataset from your own work, which would take more thought: defining the target, preventing leakage or deciding what a costly error looks like? Share your question in the comments.

Technical documentation

• Scikit-learn installation requirements
• The bundled wine dataset
• Scikit-learn: data leakage and common pitfalls
• Model evaluation and scoring

Related practical guides

• Choosing open-source AI tools
• A safe Jupyter Notebook setup
• Taking an approved AI model toward production