The Number of Triangles Whose Vertices Are at the Vertices of a Regular Octagon but None of Whose Sides Is a Side of the Octagon Is

JEE Main 2024 — 6 April, Morning Shift. Previous Year Question.

Problem

The number of triangles whose vertices are at the vertices of a regular octagon, but none of whose sides is a side of the octagon, is:

Key insight. It’s far easier to count the triangles that do share at least one side with the octagon and subtract them from the total, than to directly count only the triangles that avoid every octagon side.

Watch this explained step by step →  ·  More on the Shiwam’s Classes channel

Approach

Start with the total number of triangles formable from the octagon’s 8 vertices, with no restriction. Then count separately the triangles that share exactly one side with the octagon, and those that share exactly two sides (three consecutive vertices) — every triangle with at least one common side falls into exactly one of these two categories. Subtracting this total from the unrestricted count gives triangles with no side in common with the octagon.

Solution

Step 1 — Count all triangles with no restriction

Choosing any 3 of the octagon’s 8 vertices:

$${}^8C_3 = \frac{8\times7\times6}{3\times2\times1} = 56$$

Step 2 — Count triangles with exactly one side common to the octagon

Choose one of the octagon’s 8 sides to be a side of the triangle. The third vertex must not be adjacent to either endpoint of that chosen side — otherwise a second octagon side would also become a side of the triangle. Out of the remaining 6 vertices (excluding the 2 endpoints of the chosen side), 2 are adjacent to those endpoints, leaving:

$$8-2-2 = 4 \text{ valid choices for the third vertex}$$

So the number of triangles with exactly one common side is:

$$8\times4 = 32$$

Step 3 — Count triangles with exactly two sides common to the octagon

A triangle shares two sides with the octagon exactly when its three vertices are three consecutive vertices of the octagon (so both sides between them are octagon sides). There are exactly 8 such triples — one starting at each vertex:

$$8$$

Step 4 — Add the two counts

$$\text{Triangles with at least one common side} = 32+8 = 40$$

Step 5 — Subtract from the total

$$\text{Triangles with no common side} = 56-40 = 16$$

Answer

Common mistakes

• Forgetting to exclude the vertices adjacent to the chosen side in Step 2. Choosing the third vertex from all 6 remaining vertices (rather than the 4 that avoid creating a second common side) overcounts triangles that actually have two sides in common with the octagon, not one — this is exactly what the “3 consecutive vertices” case is meant to count separately.
• Double-counting the two-common-side triangles. Each set of 3 consecutive vertices genuinely has two octagon sides, so it’s already excluded from the “exactly one side” count in Step 2 — adding it as its own separate category in Step 4, rather than trying to fold it into Step 2’s count, keeps every triangle counted exactly once.
• Using this method’s numbers for a different-sized polygon without adjusting. The general formula from this approach is $\dfrac{n(n-4)(n-5)}6$ for an $n$-sided regular polygon — plugging in $n=8$ gives $16$, but the same formula with a different $n$, like a decagon, gives a different result.

Practise next

• Find the number of triangles whose vertices are at the vertices of a regular decagon (10 sides), but none of whose sides is a side of the decagon.