BSPX is a format originally invented by Tonik, I believe, and is already implemented in both ezQuake and FTE.
It is not so much a format itself, but more of an extensible way to shove additional lumps within an existing BSP.
Typically these additional lumps will provide supplemental information or replace existing info.
BSPX itself can logically be applied to the BSP file format of Quake, Quake II, or Quake III.

BSPX itself can be defined in just the following two structures:
typedef struct {
	char lumpname[24]; // up to 23 chars, zero-padded
	int fileofs;  // from file start
	int filelen;
} bspx_lump_t;
typedef struct {
	char id[4];  // 'BSPX'
	int numlumps;
	bspx_lump_t lumps[1];
} bspx_header_t;

The bspx_header_t struct can be found immediately following the BSP's standard header. It is only valid if the standard header specifies no lump as starting at that location, and if the magic id matches.
The engine can then walk the lump list looking for lumps that it recognises. Unknown lumps MUST be ignored.



These lumps are currently defined:

RGBLIGHTING:
(applies to fte, ezquake, qss)
This is equivelent to the information stored in a .lit file (sans header), and must contain the same number of samples as the lightmap lump would normally contain, because it doesn't make much sense otherwise.
Presence of this lump permits omitting the regular mono lightmap to save space, but doing so will harm compatibility.

LIGHTING_E5BGR9:
(applies to fte, parsed by qss only as a fallback)
This is a more advanced alternative to RGBLIGHTING.
Each luxel is a E5BGR9 int32 packed value (ie: on little-endian machines, the exponent is the high 5 bits), resulting in what is effectively a memory-efficient floating point rgb value.
This lightmap format virtually removes all oversaturation limits, and promises greater precision in dark areas too.
This format is directly supported on ALL OpenGL[ES] 3.0+ gpus (aka: GL_EXT_texture_shared_exponent).
As a floating point format, a logical value of 1.0 is considered as identity (instead of 255 being an [overbright] multiplier of 2.0).
Lighting values are always assumed to be linear.

LIGHTINGDIR:
(applies to fte)
This lump contains averaged surface-space light directions (read: deluxemap), equivelent to fte's .lux file, or dp's .dlit files (sans header).
(as unorm values, these need to be biased before use).
If bumpmaps or specular maps are not available then the effects of this may not be significant/noticable.

LMSHIFT:
(applies to fte, qss)
This is a series of per-surface bytes. Each byte provides the (1<<shift) ratio of texels-per-luxel of the corresponding surface.
Note that the engine's per-surface lightmap size limit still applies, but any engine that supports LMSHIFT must support up to 256 luxels in either axis (note that 1 luxel is reserved for edges, so a shift of 0 requires that surfaces be subdivided to at least a 255qu boundary while a shift of 4 reduces the required subdivisions on an infinitely sized surface by a factor of 256, or to a max extent of 4080qu).
As a result, the subdivide size argument of qbsp can limit the minimum shift value (maximum lightmap scale).

LMOFFSET:
(applies to fte, qss)
This replaces the lightmap offset value of the surface lumps. Should only be used in conjunction with LMSHIFT.
This allows a bsp to contain both scaled surfaces and unscaled ones, without looking broken in engines that use either mode.

LMSTYLE:
(applies to fte, qss)
This replaces the four lightstyle indexes of each surface lump entry. Normally only makes sense when used in conjunction with LMSHIFT.

LMSTYLE16:
(applies to fte, qss)
An upgrade for the LMSTYLE lump, potentially giving up to 65k lightstyles.
Additionally, the number of styles per face is variable, and must be inferred by the lump size vs surface count.
Both of these tweeks make it useful even without LMSHIFT.

VERTEXNORMALS:
(applies to fte)
This lump contains a series of vec3_t values. One per vertex lump entry.
Surfaces with a texinfo flag of 0x800 will use this lump in order to determine vertex normals, otherwise they will use their plane's normal.
These normals allow rtlights to respond to curved forms, while the flag prevents the need for excessive vertex+edge counts.
Verticies that are not part of any 0x800 surface will not be read, and will usually hold values of 0 (if only because it compresses better).

BRUSHLIST:
(applies to fte)
Engines that utilise this lump will no longer need to use hull-based collisions.
struct {
	unsigned int ver = 1;
	unsigned int modelnum;		//inline model number. 0 for world, obviously.
	unsigned int numbrushes;	//size of following array.
	unsigned int numplanes;		//total count. for validation.
	struct
	{
		vec_t mins;
		vec_t maxs;
		signed short contents;
		unsigned short numplanes;
		{
			vec3_t 	normal;
			float 	dist;
		} planes[numplanes];
	} brush[numbrushes];
} permodel[];
A submodel could be an illusionary model with no solid surfaces. Such a model will become non-solid if there is a BRUSHLIST lump that does not name the submodel.
Axial planes MUST NOT be written - they will be inferred from the brush's mins+maxs. This guarentees that brush expansion does not extend points beyond the brush itself, and saves size on maps made by lazy people.
Contents values are equal to Quake's normal CONTENT_FOO values. This outbreak of lack of imagination is brought to you by a desire to avoid politics about content masks.
CONTENTS_EMPTY	= -1	//an error. pointless.
CONTENTS_SOLID	= -2	//regular solidity
CONTENTS_WATER	= -3	//required for pointcontents to work properly.
CONTENTS_SLIME	= -4	//really I'm only listing these for completeness. only the last two are 'new'.
CONTENTS_LAVA	= -5	//it burns!
CONTENTS_SKY	= -6	//nothing explodes here!
CONTENTS_CLIP	= -8	//borrowed from halflife, blocks only entities with size.
CONTENTS_LADDER	= -16	//borrowed from halflife
Presence of this lump permits hulls 1 and 2 to be entirely omitted from the BSP (0 is still used for rendering), but doing so will harm compatibility with engines that do not understand this (presumably a solid-but-valid tree should be written, just in case). This could boost load times.
The engine is expected to insert the brushes into the bsp's various nodes. Inserting them into the leafs is suboptimal as leaf 0 will end up containing every brush...

ENVMAP:
(applies to fte)
Generated by fte's map_findcubemaps by parsing the entities lump for env_cubemap entities.
struct
{
	vec3_t origin;
	int cubesize;
} envmaps[];
This defines the various envmaps in the bsp, for mdls to use.
envmaps will be loaded from textures/env/MAPNAME_X_Y_Z using the engine's regular cubemap rules (xyz are rounded to ints).

SURFENVMAP:
(applies to fte)
Generated by fte's map_findcubemaps by parsing the entities lump for env_cubemap entities.
struct
{
	unsigned int envmapindex;
} persurface[];
This provides each surface in the bsp with info about its most appropriate(nearest) envmap. if the index is ~0u, then no envmap is specified.


Other features:
FTE supports mipmaps with all four offsets set to 0 as external textures. Such texture will ignore gl_load24bit, treating it as always enabled. This saves space and simplifies copyright ownership without resulting in untextured maps (when the textures are actually available, anyway).

Misc Compatibility Concerns:
mipmap lump - mipmaps with all four offsets set to 0 are external textures and contain no actual data. Note that vanilla quake will glitch out if given such a texture, although its unlikely to crash.
hulls - presence of BRUSHLIST allows hulls 1 and 2 to be ommitted. This will crash vanilla glquake but not winquake, and can be reproduced with the vanilla tools too.
lightmap sizes - winquake is limited to 18*18 lightmaps. Many quake engines support up to 256*256 now (AFTER lmscale). Note that such large lightmaps are not recommended due to all the resulting texture switches.