What I Learned About Social Engineering Tradecraft from Human Hackers

Social engineering can describe their tactics but not what actually made a call work, because the skill runs on implicit pattern recognition, not the explicit script awareness training assumes. If we rely on intuitions, detection is left up to experience and chance.
DEFCON was spread out across three levels at the Las Vegas Convention Center with workshops, lectures, and demonstrations ranging from satellite hacking to maritime and application security. I spent most of my time watching the main competitors, ‘cold calls’, qualifiers, and judge feedback in between.
Four of the competitors from these year’s competitions were kind enough to sit down with me. They has a range of experience, with one newcomer spending months conducting OSINT and honing their pretext to seasoned read teamers who have done this for years.
The gap between the folk wisdom of social engineering and what the practitioners were doing is wide. While the people engaged in red teaming are not necessarily like [malicious] social engineers, nor is the CTF-based directed toward stealing credentials or compromising systems, their thought processes provide invaluable insight into how social engineering works.
How the DEFCON Competition Works
The calls happen in a sound-proof booth. They are effective - I used a similar set-up for my research on speech perception. For the contestant it means the audience watches while they - and the [caller] - neither see nor hear the audience.
Ethics limits the contestants. They cannot request passwords or personal information, cannot make promises that will go unfulfilled, and cannot use pretexts that impersonate law enforcement or any legally liable entity. No one is harmed to score a flag. This is a demonstration to help individuals and companies learn.
The formats vary in how much preparation they allow. Cold calls give almost none: the competitor draws three flags rated easy, medium, and hard, and has a few minutes to capture them. The qualifiers, new this year, gave contestants under 24 hours to gather OSINT and build a pretext. The main competition gives months.
In the main event each contestant also had to work in one audience-chosen question, deliberately odd: which superhero would you be, the strangest thing you have ever eaten? This is the real test as it is misaligned with the security questions.
Table 1. Tasks in the competition.
Thinking Like a Social Engineer
Books have been written on social engineering, with classics such as Mitnick’s classic The Art of Deception to Social Engineering: The Science of Human Hacking* and Cialdini’s psychological accounts Persuasion and Kahneman’s Thinking Fast and Slow. As someone who has studied expert clinician decision-making, I wanted to see how each of these contestant approached the task of social engineering.
The Knowledge is Largely Implicit. One of the questions I asked to each of the competitors was how they knew what they knew. As a largely verbal task, social engineering would seem to require verbalizable knowledge. Instead, much of it appears implicit.
Their knowledge can be divided into explicit and implicit. Explicit knowledge is what a social engineer is aware of in the moment: the tactics, principle, and pretexts. While informative, there what makes it work in the moment is much deeper. The competitors gave me very similar answers: in the moment, you have to adapt to maintain trust and the flow. The months of OSINT gave them a clear mental model of the organization on the other end and the kind of person they would speak to. But in the moment, all the preparation and deliberation are secondary to the situation.
The descriptive language is captured by the judges. These are the true experts. Their explanations of technique are eloquent and clear, informed by the literature. But the judges didn’t always know themselves what would work. After one cold call, a judged told one competitor that a student pretext never works. In the next call, it did.
Some of this is luck: who is on the other end matters as much as the pretext. But the ‘art’ of social engineering really hides the implicit knowledge that makes social engineering work. This comes from social competence, practice, and feedback.
Congruency is the Key to Success. Pretexts are the explicit core of social engineering. They consist of the social cues and causal narratives that define real situations. Building a pretext requires developing an explicit understanding of how a company does or could function or who does or could work there. If OSINT or reconnaissance are not enough, ambiguity allows the target to fill in the blanks.
The key is congruency: the features that the social engineer and employee use to understand the world must overlap. As the two drift, this deviance becomes a ‘bad feeling’. An insider who knows the workflows, the transaction types, and the project names hears the wrong term at once and marks the caller as an outsider.
This is social script substitution: Social engineers want to avoid affective uncertainty, leading to doubt. They must use familiar cues to activate business transactions and social situations in a victims memory.

Figure 1. Negative affect (unease) increases in new situations and peaks at the boundary between social and business scripts, where neither script fully consistent. It falls in normal social and business contexts, where a pretext that stays within the respective script promotes no unease.
The “IT security audit pretext” was the most common at the competition, partly because of the CTF format. Its questions belong to security-awareness training, and few people care about that material enough to learn it well.
There is also a trick question. At the end, competitors try to ask something discrepant. The trick is to seed relatable, causal content early. If I ask formal questions throughout and then drop an informal one, it jars. If I build informal framing all along, it lays the ground for bypassing normal mental verification.
Social Engineering is About Engagement. Victims are not passive. They are active participants. In fact, the more active they are, they more the relationship can build over time.
A focus on rapport building ignores a simple fact: the targets did not always like the caller. Commitments are what counts. A person wants to be consistent within their job and within the moment. Part of their job is to avoid unnecessary or insecure disclosure. They must challenge and avoid compliance. But they must also help customers and people within their organization.
In one demonstration, the victim is immediately suspicious when the task is described. They seem to comply because the question seems innocuous. Each question is recieved with suspicion, and replied to regardless. Once they are told its the end, the victim responds in a pleasant tone.
For anyone who knows psychology, this should not be surprising. Attitudes are only weak predictors of behavior. A victim can maintain suspicions while letting more familiar, dominant response take the wheel. The context is critical.
We Cannot Rely on Human Intuition to Detect Social Engineering
The barriers to entry for social engineering might seem low, but there’s a signifiant gap between an artful novice who can improvise and a expert social engineer. Stopping social engineering requires an comprehensive understanding of the tactics and techniques that are the most successful.
As long as that knowledge remains implicit, defenses will be based on the defender having the right experience. If IT support and help desk agents do not have this, subtle manipulation will go unnoticed. Defending consumers and organizations requires that we understand the patterns of social engineering to detect them.
Social engineering detection requires years of training. Red team testing is useful, but has its limits: the more tests that occur, the more business processes are disrupted. If employees cannot be trained to hold a multi-cue pattern in mind, we must develop another means of detection: technologies that analyze sequences and timing errors a human misses under pressure, paired with procedures that do not depend on how the call feels.
What is feature engineering
In practice, feature engineering is both science and a bit of witchcraft. It often involves both iteration and experimentation to uncover hidden patterns and relationships within the data. For instance, a data scientist might transform raw sales data into features such as average purchase value, purchase frequency, or customer lifetime value, which can significantly boost the performance of a churn prediction model. By thoughtfully engineering features, practitioners can provide machine learning models with the most informative inputs, ultimately leading to better accuracy and more robust predictions.
What’s more?
- Incorporate more and more data sources
- Feature engineering platform
What is data engineering
As we mentioned above, feature engineering is certainly a subset of data engineering. It involves the ingestion of data from a source, applying a series of transformations, and making the final result available to be queried by a model for training purposes. You can construct feature engineering pipelines to resemble data engineering pipelines, having schedules, specific source and sink destinations, and availability for querying. However, this configuration would only really apply once you have surpassed the experimentation stage and determined a need for a consistent flow of new feature data.
What is feature engineering

1. Functions
Functionally, there is nothing to differentiate data vs features - data points (link). Where feature engineering and data engineering really differ is in the objectives and motivations for constructing the pipelines. In general, data engineering serves a broader, more unified purpose than feature engineering. Data engineering platforms are constructed to be flexible and universal, ingesting various types and sources of data into a unified storage location where any number of transformations and use cases can be applied. The intent of a well constructed fact table or gold layer in a data lake is to provide a single source of truth that answers many different questions, produces many reports, and can be consumed by many downstream customers.
2. Practise
And in practice, an organization’s data engineering team will be responsible for the curation and maintenance of all data pipelines, not just those that relate to machine learning. These pipelines may power BI dashboards used by C-Suite, auditing reports that feed payroll, or event logs that show a user’s history of actions within the application.
Feature engineering, on the other hand, serves a specific purpose, finding the tailored inputs and columns that will generate the best predictive results for a machine learning model. Data scientists and machine learning engineers are not tasked with developing a universal data model that will ingest all data points throughout an organization, they just need to select, curate, and clean the data needed to power their models.
3. Machine learning
Now, as machine learning teams grow and begin to incorporate more and more data sources into their models, their feature engineering platform may start to resemble a larger data engineering platform in the tools and methodologies they employ. But, the intent is not to establish flexible data models that can be used throughout the organization - it is simply to power their machine learning models.






